Functional tests take a long time

From CitconWiki
Revision as of 01:06, 10 April 2008 by Jtf (talk | contribs) (adding my rough draft notes from the session)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigationJump to search

Long Running Functional Tests

These are the few notes I took from the "long running functional tests" discussions.

Problems

  • Complete features take 1 day
  • Functional test takes 15 hours

Alternatives, Risks, and Trade-offs

  • Parallel testing vs. pipelined testing
    • Compile
    • Fast unit tests
    • Slow unit tests
    • Functional tests
  • Incremental feedback during test runs
    • Show failures sooner, but
    • Does not typically lead to stopping the tests because we want to know all the results from that set of code


Alternate set of incomplete notes:

What do I do if my functional tests take 5 hours?

4 hour cycle vs. 5 minutes

(ci-machismo: "mine is smaller than yours")

some tests we run all the time and some that we don't

order of magnitude & build pipeline tradeoff: 5 minutes vs. 50 20 minutes vs. 30

long running test suite but reporting of status after each suite (separate emails?)

re-order decomposable steps

what tools are people using to do their functional tests?

new process: added qa, added release management. developers aren't responsive to breaks in the QA builds. often there is disagreement about what the customer really wants, what the problems are. mini-waterfall labeled as Agile. requirements sprint, design sprint, coding sprint, testing sprint.

Retrieved from "http://citconf.com/wiki/index.php?title=Functional_tests_take_a_long_time&oldid=5744"