Difference between revisions of "Functional tests take a long time"
MarkEWaite (talk | contribs) (New page: Long Running Functional Tests These are the few notes I took from the "long running functional tests" discussions. Problems * Complete features take 1 day * Functional test takes 15 hou...) |
(adding my rough draft notes from the session) |
||
| (One intermediate revision by one other user not shown) | |||
| Line 3: | Line 3: | ||
These are the few notes I took from the "long running functional tests" discussions. | These are the few notes I took from the "long running functional tests" discussions. | ||
| − | Problems | + | '''Problems''' |
* Complete features take 1 day | * Complete features take 1 day | ||
* Functional test takes 15 hours | * Functional test takes 15 hours | ||
| − | Alternatives, Risks, and Trade-offs | + | '''Alternatives, Risks, and Trade-offs''' |
* Parallel testing vs. pipelined testing | * Parallel testing vs. pipelined testing | ||
| Line 19: | Line 19: | ||
** Show failures sooner, but | ** Show failures sooner, but | ||
** Does not typically lead to stopping the tests because we want to know all the results from that set of code | ** Does not typically lead to stopping the tests because we want to know all the results from that set of code | ||
| + | |||
| + | |||
| + | Alternate set of incomplete notes: | ||
| + | |||
| + | What do I do if my functional tests take 5 hours? | ||
| + | |||
| + | 4 hour cycle vs. 5 minutes | ||
| + | |||
| + | (ci-machismo: "mine is smaller than yours") | ||
| + | |||
| + | some tests we run all the time and some that we don't | ||
| + | |||
| + | order of magnitude & build pipeline tradeoff: | ||
| + | 5 minutes vs. 50 | ||
| + | 20 minutes vs. 30 | ||
| + | |||
| + | long running test suite but reporting of status after each suite (separate emails?) | ||
| + | |||
| + | re-order decomposable steps | ||
| + | |||
| + | what tools are people using to do their functional tests? | ||
| + | |||
| + | new process: added qa, added release management. developers aren't responsive to breaks in the QA builds. often there is disagreement about what the customer really wants, what the problems are. mini-waterfall labeled as Agile. requirements sprint, design sprint, coding sprint, testing sprint. | ||
Latest revision as of 01:06, 10 April 2008
Long Running Functional Tests
These are the few notes I took from the "long running functional tests" discussions.
Problems
- Complete features take 1 day
- Functional test takes 15 hours
Alternatives, Risks, and Trade-offs
- Parallel testing vs. pipelined testing
- Compile
- Fast unit tests
- Slow unit tests
- Functional tests
- Incremental feedback during test runs
- Show failures sooner, but
- Does not typically lead to stopping the tests because we want to know all the results from that set of code
Alternate set of incomplete notes:
What do I do if my functional tests take 5 hours?
4 hour cycle vs. 5 minutes
(ci-machismo: "mine is smaller than yours")
some tests we run all the time and some that we don't
order of magnitude & build pipeline tradeoff: 5 minutes vs. 50 20 minutes vs. 30
long running test suite but reporting of status after each suite (separate emails?)
re-order decomposable steps
what tools are people using to do their functional tests?
new process: added qa, added release management. developers aren't responsive to breaks in the QA builds. often there is disagreement about what the customer really wants, what the problems are. mini-waterfall labeled as Agile. requirements sprint, design sprint, coding sprint, testing sprint.
