Test Triage and Intermittent Test techniques

From CitconWiki
Jump to navigationJump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

Attendance

  • Tyler @macetw
  • Emil
  • @RobPark

Test Triage Techniques

Often tests appear without clear reason, many commits into a single build. Unreliable machines or frameworks, how do we respond?

Approaches:

  • Not enough information in the log
  • Limit smoke test time

1 team might be dedicated to triaging

  • Build sheriff
  • Assign person responsible
  • File ticket

Assumed: Does everyone care that a build is red? Is a build/test cycle easy enough to reproduce?

Intermittent Failures

CI tool (TeamCity, Jenkins plugin) will handle assignment of an issue, a particular failing test, and there would be an owner of a certain issue, even if not always-failing.

Blame on load average, up-of-timeout-wait, but no real fix.

Approaches:

  • Ticket made of while a build intermittently failed.
  • Assign test to quarantine
    • Management approval required
    • Analysis of time-management
    • Stuff moves gradually out of quarantine

Sidebar: Google test on the toilet mental note, for macetw, we need more code-literature in our bathroom

One approach:

  • Remove from test running
    • Would affect code coverage numbers
    • Not all cultures
  • If still present in code 90 days later, may choose delete test scenario


Sidebar: If high defect rate, but also high code coverage, tests may not be testing the right kinds of things