Test Triage and Intermittent Test techniques

From CitconWiki
Revision as of 08:51, 24 August 2013 by Macetw (talk | contribs)
Jump to navigationJump to search

Attendance

  • Tyler @macetw
  • Emil
  • @RobPark

Test Triage Techniques

Often tests appear without clear reason, many commits into a single build. Unreliable machines or frameworks, how do we respond?

Approaches:

  • Not enough information in the log
  • Limit smoke test time

1 team might be dedicated to triaging

  • Build sheriff
  • Assign person responsible
  • File ticket

Assumed: Does everyone care that a build is red? Is a build/test cycle easy enough to reproduce?

Intermittent Failures

CI tool (TeamCity, Jenkins plugin) will handle assignment of an issue, a particular failing test, and there would be an owner of a certain issue, even if not always-failing.

Blame on load average, up-of-timeout-wait, but no real fix.

Approaches:

  • Ticket made of while a build intermittently failed.
  • Assign test to quarantine
    • Management approval required
    • Analysis of time-management
    • Stuff moves gradually out of quarantine


Sidebar: Google test on the toilet mental note, for macetw, we need more code-literature in our bathroom