Nigel.charman at 01:15, 9 February 2013

2013-02-09T01:15:24Z

Nigel.charman: Created page with "No means No - how to keep testing failures meaningful in CI. Slow tests. Flaky tests. Tests that change. Katrain Edgar Devs writing unit tests Testers writing integration te..."

2013-02-09T01:11:23Z

Created page with "No means No - how to keep testing failures meaningful in CI. Slow tests. Flaky tests. Tests that change. Katrain Edgar Devs writing unit tests Testers writing integration te..."

New page

No means No - how to keep testing failures meaningful in CI. Slow tests. Flaky tests. Tests that change.

Katrain Edgar

Devs writing unit tests
Testers writing integration tests
Devs multiple check-ins per day.
Difficult to get integration tests meaningful to testers
2 hour test suite:
Bad code, copy and paste
Moved setup steps out of Selenium

got down to half hour
Fragile – needed to change sleeps to waits

Because tester had ownership, tests would break and testers would have to constant stream of red builds. Devs stopped paying attention to red builds.

Integration tests only helpful to teste r, became same interpretation as in manual environment.

Loop of death - different reasons for tests to go red each time
Green light once a week, celebrated with coffee, acknowledged as a problem (“squirrel dance” at TIM)

Andrew – same problem but worse, not part of deployment pipeline
5 devs, 2 testers – needs more testers than usual

Martin – integration tests owned by devs, testing team test from external interfaces, large amount of manual testing, embedded radio systems, automated environments with hand-held units, tests call quality etc.
Same problem with Jenkins turning red and slow feedback time
Looking to run smoke test on each build, then less frequent or nightly build
Painful to install stuff onto device
Dev checkin – 30 minutes for CI tests to complete, some tests only nightly

Julian – canary tests for risky areas,

How to tackle continual red builds?
Potential for doing a “hearts and minds”
Katrina tried Gold star charts, which was a good motivation for devs
Also got devs writing the tests

Ward – failures are personal, need to make it fun, if you're not writing bugs, you're not writing code

HTML tests are always going to be flaky – :because browsers suck”, write tests at service layer, subcutaneous testing
Devs already had responsibility for the quick tests
Would you throw away GUI tests? Tend to be repetitive, need to refactor down

False negatives -
Claim plugin – assigned “cake points”, if you didn't claim you had to bring in cake. Plus cafe bonuses

Jenkins game plugin – useful to start people getting interested in it, but could lead to bad behaviour (eg. Checking in meaningless tests to get points)
Stop the line on broken builds
Make a developer responsible for checking the build and doing triage of failures – can make you feel crap always have to go back to same person
What was better – picking on one person or stopping whole team

Reverting check-ins
Validated merge plugin
Git plugin – merge to branch on successful build
Gerrit

Source code management
10 teams checking in on branches then merging to trunk. Teams have to wait when trunk is broken.

Visibility of breakage
build radiator
USB tower of LEDs that showed breakages

Build radiator also showed message of the day, jokes etc to act as central source of information

Have to slow down before you speed up.
Not doing CI if you're not stopping when it breaks

Look at definition of done criteria – can't claim points until its green

CD is powerful – can't deploy until working

ATDD – check-in of incomplete features, use Pending/Expected to fail flag on acceptance tests while developing feature and checking in successul unit tests

Pushback from devs on creating and running integration tests. Breaks “flow”.

Acceptance tests shouldn't fail if sufficient testing at a lower level.

Test on own machine – can pass, then still fail on build server due to environmental issues

By time of failure, multiple commits have been picked up so hard to ascertain blame. Potential changes – slowing down commits, concurrent builds, spin-up multiple environments in the cloud to run tests that require environment

Devs unwilling to run tests, needs extra environmental setup

Jan - 30 minutes integration test time. Devs run tests over lunch or at end of day. Commit every 6-8 working hours.
Daphne – use Git for tiny commits.

Delete integration tests that never fail.

Devs saw as someone else's code, tester owned. Potentially having the tester pair with developer may have resulted in shared ownership. Co-location helps.

Make it fun – devs will stick around longer and do extra stuff.

Team ownership of broken builds.

Silos can cause friction – eg. Different reporting lines for devs and testers, turf war.
Needs buy-in from management, and focus on better working relationships.

Coding dojos – get team working together on shared goal that's not production code, can use CI approach and ensure CI principles are followed

In large org, having meeting 2-3 times a week across scrum of scrums helps get understanding of what is being committed and less broken builds

@@ Line 1: / Line 1: @@
 No means No - how to keep testing failures meaningful in CI. Slow tests. Flaky tests. Tests that change.
-Katrain Edgar
+Katrina Edgar
 Devs writing unit tests
@@ Line 111: / Line 111: @@
 Silos can cause friction – eg. Different reporting lines for devs and testers, turf war.
 Needs buy-in from management, and focus on better working relationships.
 Coding dojos – get team working together on shared goal that's not production code, can use CI approach and ensure CI principles are followed
 In large org, having meeting 2-3 times a week across scrum of scrums helps get understanding of what is being committed and less broken builds

NoMeansNo - Revision history

Nigel.charman at 01:15, 9 February 2013

Nigel.charman: Created page with "No means No - how to keep testing failures meaningful in CI. Slow tests. Flaky tests. Tests that change. Katrain Edgar Devs writing unit tests Testers writing integration te..."