Functional tests take a long time
Long Running Functional Tests
These are the few notes I took from the "long running functional tests" discussions.
- Complete features take 1 day
- Functional test takes 15 hours
Alternatives, Risks, and Trade-offs
- Parallel testing vs. pipelined testing
- Fast unit tests
- Slow unit tests
- Functional tests
- Incremental feedback during test runs
- Show failures sooner, but
- Does not typically lead to stopping the tests because we want to know all the results from that set of code
Alternate set of incomplete notes:
What do I do if my functional tests take 5 hours?
4 hour cycle vs. 5 minutes
(ci-machismo: "mine is smaller than yours")
some tests we run all the time and some that we don't
order of magnitude & build pipeline tradeoff: 5 minutes vs. 50 20 minutes vs. 30
long running test suite but reporting of status after each suite (separate emails?)
re-order decomposable steps
what tools are people using to do their functional tests?
new process: added qa, added release management. developers aren't responsive to breaks in the QA builds. often there is disagreement about what the customer really wants, what the problems are. mini-waterfall labeled as Agile. requirements sprint, design sprint, coding sprint, testing sprint.