Continuous performance test

I cant really do this excellent session justice so hopefully somebody will come along and update these notes!

This is what I took away:

Performance testers as first class citizens

There was a mixed group of performance test architects (!), developers, testers.

The performance testers felt that they were misunderstood and poorly supported sometimes by the developers.

In an agile environment most people agreed that when constructing stories you must create stories that feature the performance test analyst as a stakeholder explicitly. E.g.

The performance test analyst would like to be able to view real time measures of the latency requests to each type of downstream system so that they can identify if there is a bottleneck and tune accordingly.

There was also a feeling from several developers that the performance testers sometimes were overly protective of their knowledge (for various reasons) and did not want to involve developers or have their contribution in any way automated. This reflects my own experience on several projects.

[What follows was never discussed in the session apologies but I think is relevant].

For example, I was involved in high volume, high concurrency, low latency mobile application. The dedicated perf test team insisted on working in isolation. At great expense (license costs for tools, consultancy fees) they produced a test suite. I insisted on being closely involved as the tech arch with delivery responsibility. I found that the perf tests being executed did not mirror the user stories supported by the system, did not stress the parts of the architecture that had been identified as potentially weak (e.g. do not spend 80% of your effort load testing the download of a resource which is actually static and ignore the resource which is protected by a call to a concurrency constrained authentication system). I also found the scripting tools themselves were a blast from the past. They were as complex as any Java code we ran but were written without structure or rigor in an obtuse dialect of pseudo-C. I rebuilt the load tests from scratch using a simple Open Source tool that developers were able to run all the time and maintain themselves. Our new load tests were empirically proved fit for purpose. The load testing team were made redundant and replaced with a combination of developers and operations staff (in true devops style).

There is a difference between component perf testing and system testing

We struggled for a while with terminology. Developers were keen to test component and measure that components relative performance as development progressed. They were also keen not to optimize early and produce end up fixing something that when the system was exercised holistically, was irrelevant.

Perf testers were keen that we focus on complete end to end system testing as at the end of the day, that was the only measure that actually counted and that could be relied on to reflect reality.

I think in summary we were all in violent agreement.

Only one person appeared to be really doing continuous performance testing

Many people in the room were doing performance testing. Several people had done performance testing early and often but still essentially executed manual tests with manual analysis. One person was lucky enough to have a perf test environment (in EC2) that was identical to production. That environment was torn down and recreated every day, several perf tests based on use cases executed and the results graphed. Nirvana!

However, that system was not yet live. Several of the group questioned how the perf tests would evolve to reflect a post-launch database and wheter the use cases would be reworked to reflect real usage of the system.

The group member with the perfect environment admitted that even with this toy he was still going to do a perf test exercise towards the end of development.

I don't think we addressed Arnaud's point!

The submitter of the session was looking for something different to what we eventually discussed I suspect. He wanted some instantaneous measure of the cost of execution of a piece of code, possibly from static analysis of code. He felt that this was the only true way to really progress high performance code development. I have not done his suggestions justice at all here - sorry!