Virtualisation in CI

Lots of interest in saving on hardware, lots of variation in the tools used.

Purposes

One major purpose is for platform testing. An example was given of having 117 combinations of Windows environments to cover every client configuration. This kind of testing could be done with real servers. Another major purpose is how to create a set of systems and pull them into one canned environment.

Reasons to use VM in CI:

• Test early
• Cheaper
• Releasing more often

It's particularly useful to have a reliable baseline and virtualisation is almost a requirement for implementing this. An example of unreliable environment was when the build is affected by changes in the build tool configuration. Virtualisation gives base lines gives you more confidence in your test results.

Tools Discussion

• Soyuz allows you to buld your own personal package repositories and build you own images on your package and private repositories that can be installed on the image.
• Setting up your own package repositories is quite easy.
• What is a method of preparing a snapshot so that when it boots there is nothing else to be done, eg no downloads needed? VMLogic has two top-level concepts within LabManager which allow you to install packages in sequence, then save the job as a running virtual machine. Starting jobs save time.

Lessons Learnt

• Oracle is building DB4 in a mutli-processor setup, making it difficult to run under a virtualised environment.
• Solaris containers are very useful for troubleshooting integration testing where the apps require different ports.
• Anti-virus scanners are extremely difficult to manage on the image, eg preventing some machines from updating, and can have a significant effecto n the behaviour of the application. Snapshotting can help here.
• Deploying updates to the services running inside the image can very difficult.
• VM-based testing is more efficient when broken up into suites rather than run in a single build. Build times become more reliable.

Observations

• The build tool 'should' manage the configuration, but the CI tool needs to get involved to make the build run inside the environment, especially in the situation of running cross-image testing.
• CMMI has a requirement that the developer must be separate from the tester at a functional tester level, though not necessarily at a unit test level.
• Virtualisation is useful for provisioning and to increase capacity. There is not yet a common set of tools for doing this kind of work.
• Would be great to be able to build VMWare environments by cascading configuration packages, a la CSS for VMWare configuration.

Testing

• Testing may start to move towards statistical selection of tests. Not all languages allow this, functional languages are better but some OO are moving towards this. Enables test coverage without needing 100% coverage. Concept is similar to test optimization, in that it examines complexity to work out which areas most need testing as a result of a given set of changes.
• Test reordering to bring risky tests forward.
• Example given in federal government of having 400 testers in per quater for a release, plus 100 permanent testers, to manage testing across all projects.