CitconWiki - User contributions [en]

Continuous performance test

2010-11-19T21:24:42Z

Anderew: New page: I cant really do this excellent session justice so hopefully somebody will come along and update these notes! This is what I took away: == Performance testers as first class citizens == ...

I cant really do this excellent session justice so hopefully somebody will come along and update these notes!

This is what I took away:

== Performance testers as first class citizens ==

There was a mixed group of performance test architects (!), developers, testers.

The performance testers felt that they were misunderstood and poorly supported sometimes by the developers.

In an agile environment most people agreed that when constructing stories you must create stories that feature the performance test analyst as a stakeholder explicitly. E.g.

''The performance test analyst would like to be able to view real time measures of the latency requests to each type of downstream system so that they can identify if there is a bottleneck and tune accordingly.''

There was also a feeling from several developers that the performance testers sometimes were overly protective of their knowledge (for various reasons) and did not want to involve developers or have their contribution in any way automated. This reflects my own experience on several projects.

[What follows was never discussed in the session apologies but I think is relevant].

For example, I was involved in high volume, high concurrency, low latency mobile application. The dedicated perf test team insisted on working in isolation. At great expense (license costs for tools, consultancy fees) they produced a test suite. I insisted on being closely involved as the tech arch with delivery responsibility. I found that the perf tests being executed did not mirror the user stories supported by the system, did not stress the parts of the architecture that had been identified as potentially weak (e.g. do not spend 80% of your effort load testing the download of a resource which is actually static and ignore the resource which is protected by a call to a concurrency constrained authentication system). I also found the scripting tools themselves were a blast from the past. They were as complex as any Java code we ran but were written without structure or rigor in an obtuse dialect of pseudo-C. I rebuilt the load tests from scratch using a simple Open Source tool that developers were able to run all the time and maintain themselves. Our new load tests were empirically proved fit for purpose. The load testing team were made redundant and replaced with a combination of developers and operations staff (in true devops style).

== There is a difference between component perf testing and system testing ==

We struggled for a while with terminology. Developers were keen to test component and measure that components relative performance as development progressed. They were also keen not to optimize early and produce end up fixing something that when the system was exercised holistically, was irrelevant.

Perf testers were keen that we focus on complete end to end system testing as at the end of the day, that was the only measure that actually counted and that could be relied on to reflect reality.

I think in summary we were all in violent agreement.

== Only one person appeared to be really doing continuous performance testing ==

Many people in the room were doing performance testing. Several people had done performance testing early and often but still essentially executed manual tests with manual analysis. One person was lucky enough to have a perf test environment (in EC2) that was identical to production. That environment was torn down and recreated every day, several perf tests based on use cases executed and the results graphed. Nirvana!

However, that system was not yet live. Several of the group questioned how the perf tests would evolve to reflect a post-launch database and wheter the use cases would be reworked to reflect real usage of the system.

The group member with the perfect environment admitted that even with this toy he was still going to do a perf test exercise towards the end of development.

== I don't think we addressed Arnaud's point! ==

The submitter of the session was looking for something different to what we eventually discussed I suspect. He wanted some instantaneous measure of the cost of execution of a piece of code, possibly from static analysis of code. He felt that this was the only true way to really progress high performance code development. I have not done his suggestions justice at all here - sorry!

CITCONEurope2010Sessions

2010-11-19T20:52:41Z

Anderew: /* 11:15 - 12:15 Sessions */

===10:00-11:00 Sessions===
[[Large Scale CI]]

[[Pairing Techniques]]

[[Performance Testing in-the-small]]

[[Database testing and deployment]]

[[Scaling agile]]

===11:15 - 12:15 Sessions===
[[Managing Multiple Dependencies]]

[[Narrative Framework]]

[[Continuous performance test]]

===14:00 - 15:00 Sessions===
[[Long Term Value of Acceptance Tests]]

[[Overcoming Organisational Defensiveness]]

=== 15:15-16:15 Sessions ===
[[Continuous Deployment]]

[[.NET CI Stack]]

[[Using KPIs/Getting a data-driven org.]]

=== 16:30-17:30 Sessions ===

[[MultipleClientDeployment]]

[[Beyond Basic TDD]]

Scaling agile

2010-11-19T20:50:55Z

Anderew: New page: Attended by a small group from various backgrounds but everybody was practicing agile AND had experienced scaling up (not much CI in this session). == Distributed Agile == Several partic...

Attended by a small group from various backgrounds but everybody was practicing agile AND had experienced scaling up (not much CI in this session).

== Distributed Agile ==

Several participants described described the issues when running a distributed team across several time zones. These were mostly practical issues and we discussed pragmatic solutions that had been tried.

One had a team with members in Vietnam and Reading. They managed to find a time of days where both teams were working (or at least awake!) but ran into problems as one team was finishing day X of the sprint whilst the other was just starting. They had had confusion around reporting progress on tasks and filling in the turndown. Several others in the group reported similar issues when the timezone difference was significant (usually transatlantic or to India). The consensus seemed to be that these were teething issues and that once an approach had been agreed the problem went away.

When dealing with a distributed team everybody agreed that everybody had to use the same medium for communication. Where one part of the team was sitting in close proximity with a number of others working remotely they had tried having a task board which some of the group could see and others used a webcam to watch. This did not work well as it always put somebody at disadvantage and lead to that person or persons not participating fully. Everyone (I think - my notes a vague!) agreed that if one part of the team was remote then everybody had to use headsets and an electronic task board. Communication channels had to be homogenous.

We discussed the practical issues of scheduling a demo when the team and various stakeholders are distributed around the planet. Everybody seemed to use Webex or some other desktop sharing style application. One member described their procedure where the scrum master would schedule a 15 minute preparation period before the demo where they would ring people up and get them connected. This sounded like a big investment but from my own experience is a great idea as it avoids a fifteen minute delay eating into the start of the demo.

== Scaling up ==

We talked about scaling up agile teams. Success was reported by a group member who had commissioned a distributed team in eastern europe. He had flown out and run a mini-sprint with the remote team so they could experience all phases of the lifecycle, the artefacts produced and the vocabulary used. This sounded like an excellent approach to me.

Another group member used rotation where an experienced member of a team would be sent to work overseas for a significant period in order to seed a new team with the culture of the initial group in the UK. This had proved effective when creating new teams in Boston and the seeding period had been three months.

We talked about experiences one group member had had with larger agile teams. In one instance the large team had been a group of committed and experienced agilsts. They had sustained a team of fifteen people effectively. Everyone was collocated and the team took responsibility for organising themselves in such a way that the agile processes (e.g. planning, standups) did not become unwieldy.

In the second instance where the team was a mix of contractors, consultants and permies at client site a different approach was required (see [http://www.slideshare.net/stephenellliott/agile-methods-experience-report-by-a-technical-architect-andrew-rendell-valtech report] and [http://www.slideshare.net/stephenellliott/the-role-of-architect-in-an-agile-organisation-andrew-rendell slides]. Here the team had to be split to keep it effective and the architecture of the application refactored so that it better reflected the teams. One of the group pointed out that this is an interesting twist on [http://en.wikipedia.org/wiki/Conway%27s_Law Conway's law].