Difference between revisions of "Test Triage and Intermittent Test techniques"
From CitconWiki
Jump to navigationJump to search| Line 7: | Line 7: | ||
Often tests appear without clear reason, many commits into a single build. Unreliable machines or frameworks, how do we respond? | Often tests appear without clear reason, many commits into a single build. Unreliable machines or frameworks, how do we respond? | ||
| − | + | Approaches: | |
* Not enough information in the log | * Not enough information in the log | ||
| − | + | * Limit smoke test time | |
| − | |||
| − | * | ||
1 team might be dedicated to triaging | 1 team might be dedicated to triaging | ||
| Line 19: | Line 17: | ||
Assumed: Does everyone care that a build is red? Is a build/test cycle easy enough to reproduce? | Assumed: Does everyone care that a build is red? Is a build/test cycle easy enough to reproduce? | ||
| + | |||
| + | ==Intermittent Failures== | ||
| + | CI tool (TeamCity, Jenkins plugin) will handle assignment of an issue, a particular failing test, and there would be an owner of a certain issue, even if not always-failing. | ||
| + | |||
| + | Blame on load average, up-of-timeout-wait, but no real fix. | ||
| + | |||
| + | Approaches: | ||
| + | * Ticket made of while a build intermittently failed. | ||
| + | * Assign test to quarantine | ||
| + | ** Management approval required | ||
| + | ** Analysis of time-management | ||
| + | ** Stuff moves gradually out of quarantine | ||
| + | |||
| + | |||
| + | Sidebar: Google test on the toilet | ||
| + | ''mental note, for macetw, we need more code-literature in our bathroom'' | ||
Revision as of 07:51, 24 August 2013
Attendance
- Tyler @macetw
- Emil
- @RobPark
Test Triage Techniques
Often tests appear without clear reason, many commits into a single build. Unreliable machines or frameworks, how do we respond?
Approaches:
- Not enough information in the log
- Limit smoke test time
1 team might be dedicated to triaging
- Build sheriff
- Assign person responsible
- File ticket
Assumed: Does everyone care that a build is red? Is a build/test cycle easy enough to reproduce?
Intermittent Failures
CI tool (TeamCity, Jenkins plugin) will handle assignment of an issue, a particular failing test, and there would be an owner of a certain issue, even if not always-failing.
Blame on load average, up-of-timeout-wait, but no real fix.
Approaches:
- Ticket made of while a build intermittently failed.
- Assign test to quarantine
- Management approval required
- Analysis of time-management
- Stuff moves gradually out of quarantine
Sidebar: Google test on the toilet
mental note, for macetw, we need more code-literature in our bathroom
