Large Scale CI
From CitconWiki
EA System
- 4 years to develop!
- end to end automated pipeline
- lots of investment
- System components
- Perforce (games use lots of binary data, Perforce very good for that type of data)
- Automation using buildforge
- not very good tracking of configuration changes in buildforge
- CruiseControl actually was very good for tracking build changes in version control
- Before buildforge, clean build took 8-10 hours
- buildforge very critical. No one able to make changes without pair programming
- 500 blades, fully virtualised, running vmware esx
- For every build, a new VM was spun up with the required software and configuration, and then shut down when the build finished
- Incredibuild, spun up buildforge instance
- Nothing special about each vm image
- netapp san storage
- netapp snapshotting (copy on write) was very good at optimising disk space. e.g. 2 different files that were mostly the same, only the diffences would be stored
- parametrised projects
- Juice: platform testing scripting language (propriatory scripting engine for games)
- No testing of "correctness" of AI, mainly worried about the fact that the game didn't crash
- Correctness mainly limited to menu interations
- Themis
- Allows juice scripts to be mapped to source code changes
- After a checkin, themis would suggest the best juice scripts to run to get the best possible coverage of code
- Allows analysis of QA coverage versus required coverage
- Reduced manual testing by huge amount
- Replay technologies
- Able to record the physical state of a piece of hardware, and replay it later for testing. Replay it on hardware with a debugger attached
- Biggest challenges
- Getting money to invest!
- Wasn't obvious to management that investing in the hardware would save money
- Management now very happy that the investment was absolutely worth it
youDevise CI Farm
- Much smaller than EA!
- Around 20 Hudson build slaves, and a similar number of slaves for Selenium Grid
- Complex configuration, needs to be simplified
- Farm nearly fully utilised during the day
- youDevise has a very large suite of browser tests
