Difference between revisions of "Test Triage and Intermittent Test techniques"

From CitconWiki
Jump to navigationJump to search
Line 38: Line 38:
 
** Not all cultures
 
** Not all cultures
 
* If still present in code 90 days later, may choose delete test framework
 
* If still present in code 90 days later, may choose delete test framework
 +
 +
 +
Sidebar: If high defect rate, but also high code coverage, tests may not be testing the right kinds of things

Revision as of 08:05, 24 August 2013

Attendance

  • Tyler @macetw
  • Emil
  • @RobPark

Test Triage Techniques

Often tests appear without clear reason, many commits into a single build. Unreliable machines or frameworks, how do we respond?

Approaches:

  • Not enough information in the log
  • Limit smoke test time

1 team might be dedicated to triaging

  • Build sheriff
  • Assign person responsible
  • File ticket

Assumed: Does everyone care that a build is red? Is a build/test cycle easy enough to reproduce?

Intermittent Failures

CI tool (TeamCity, Jenkins plugin) will handle assignment of an issue, a particular failing test, and there would be an owner of a certain issue, even if not always-failing.

Blame on load average, up-of-timeout-wait, but no real fix.

Approaches:

  • Ticket made of while a build intermittently failed.
  • Assign test to quarantine
    • Management approval required
    • Analysis of time-management
    • Stuff moves gradually out of quarantine

Sidebar: Google test on the toilet mental note, for macetw, we need more code-literature in our bathroom

One approach:

  • Remove from test running
    • Would affect code coverage numbers
    • Not all cultures
  • If still present in code 90 days later, may choose delete test framework


Sidebar: If high defect rate, but also high code coverage, tests may not be testing the right kinds of things