Long Term Value of Acceptance Tests
Does agile work?
Where's the data?
- Defect detections rate
- Number of defects caught internally vs the number of defects found by customers
- Above 80% pretty good
- Team with 99% rate, no bug in production in over 5 years
- Examle of Moving from C++ to Java
- First attempt didn't complete
- Second attempt blew up in production, reverted
- Third attempt build using detailed acceptance testing, 2 year project
- Example of Student loan company in thee US
- 2008 bond sale fell through, no money to operate (financial crisis)
- Option to completely change the way that they get money (private investors, etc)
- Because they had a good suite of acceptance tests, they were able to review their existing business workflow and go live within 2 months, able to stay in business
These are all good example of the benefits of acceptance tests
Regression testing isn't really the final goal of acceptance tests. If that's all that you aim for, you won't necesarily get the full benefits of what can be achieved
YouSwitch reported a very good cultural change, company became much more collaborative.
hugs -> gojko : a lot of people can get stuck in the UI test angle of acceptance tests, and so missing business value of the tests
gojko -> Personas can be very helpful
gojko -> acceptance tests written in terms of the UI tend to have long term maintenance problems
hugs -> the key word is "acceptance". It's monetary :-) if the customer clicks a buttton and it terns red, will they pay the invoice?
gojko -> there is no generic terms, it depends upon the company as to what term is used
gojko -> most teams use automated acceptance tests, with exploratory testing on top. No one really uses manual testing (at least, what he used). In some cases, a company didn't automate anything for the first 3 months of a project. Paul Gerrard argues that what you really need to do is to define the process of how a team is going to validate something.
gojko -> the guys who get the best long term benefit are those who put the business knowledge in the tests, which can be used as a resource for future development. These things are not really tests any more. This is more a system of documenting business processes.
gojko -> executable specfications mean that we know what the system does, and it is correct as to what the system does right now. Getting business processes documented and automatically testing against current systems is the real benefit
gojko -> the people who got the best benefits spent a lot of time organising the improving their specifications, refactoring, etc
gojko -> most of the people who ended up with systems like these got there by chance. What we need is a more deliberate way of approaching this