Long Term Value of Acceptance Tests

Does agile work?

Gojko...

Where's the data?

• Defect detections rate
• Number of defects caught internally vs the number of defects found by customers
• Above 80% pretty good
• Team with 99% rate, no bug in production in over 5 years

• Examle of Moving from C++ to Java
  • First attempt didn't complete
  • Second attempt blew up in production, reverted
  • Third attempt build using detailed acceptance testing, 2 year project

• Example of Student loan company in thee US
  • 2008 bond sale fell through, no money to operate (financial crisis)
  • Option to completely change the way that they get money (private investors, etc)
  • Because they had a good suite of acceptance tests, they were able to review their existing business workflow and go live within 2 months, able to stay in business

These are all good example of the benefits of acceptance tests

Regression testing isn't really the final goal of acceptance tests. If that's all that you aim for, you won't necesarily get the full benefits of what can be achieved

YouSwitch reported a very good cultural change, company became much more collaborative.

hugs -> gojko : a lot of people can get stuck in the UI test angle of acceptance tests, and so missing business value of the tests

gojko -> Personas can be very helpful

gojko -> acceptance tests written in terms of the UI tend to have long term maintenance problems

antony -> what people label something isn't necessarly what it is. Some people might define "acceptance tests" in terms of user experience of clicking ui elements, not really a test that defines the business benefit. It's a labelling issue

hugs -> the key word is "acceptance". It's monetary :-) if the customer clicks a buttton and it terns red, will they pay the invoice?

gojko -> there is no generic terms, it depends upon the company as to what term is used

gojko -> most teams use automated acceptance tests, with exploratory testing on top. No one really uses manual testing (at least, what he used). In some cases, a company didn't automate anything for the first 3 months of a project. Paul Gerrard argues that what you really need to do is to define the process of how a team is going to validate something.

gojko -> the guys who get the best long term benefit are those who put the business knowledge in the tests, which can be used as a resource for future development. These things are not really tests any more. This is more a system of documenting business processes.

gojko -> executable specfications mean that we know what the system does, and it is correct as to what the system does right now. Getting business processes documented and automatically testing against current systems is the real benefit

gojko -> the people who got the best benefits spent a lot of time organising the improving their specifications, refactoring, etc

gojko -> most of the people who ended up with systems like these got there by chance. What we need is a more deliberate way of approaching this