Large Scale CI

EA System

• 4 years to develop!
• end to end automated pipeline
• lots of investment
• System components
  • Perforce (games use lots of binary data, Perforce very good for that type of data)
  • Automation using buildforge
    • not very good tracking of configuration changes in buildforge
    • CruiseControl actually was very good for tracking build changes in version control
    • Before buildforge, clean build took 8-10 hours
    • buildforge very critical. No one able to make changes without pair programming
  • 500 blades, fully virtualised, running vmware esx
    • For every build, a new VM was spun up with the required software and configuration, and then shut down when the build finished
    • Incredibuild, spun up buildforge instance
    • Nothing special about each vm image
  • netapp san storage
    • netapp snapshotting (copy on write) was very good at optimising disk space. e.g. 2 different files that were mostly the same, only the diffences would be stored
  • parametrised projects
  • Juice: platform testing scripting language (propriatory scripting engine for games)
    • No testing of "correctness" of AI, mainly worried about the fact that the game didn't crash
    • Correctness mainly limited to menu interations
  • Themis
    • Allows juice scripts to be mapped to source code changes
    • After a checkin, themis would suggest the best juice scripts to run to get the best possible coverage of code
    • Allows analysis of QA coverage versus required coverage
    • Reduced manual testing by huge amount
  • Replay technologies
    • Able to record the physical state of a piece of hardware, and replay it later for testing. Replay it on hardware with a debugger attached

• Biggest challenges
  • Getting money to invest!
  • Wasn't obvious to management that investing in the hardware would save money
  • Management now very happy that the investment was absolutely worth it

youDevise CI Farm

• Much smaller than EA!
  • Around 20 Hudson build slaves, and a similar number of slaves for Selenium Grid
• Complex configuration, needs to be simplified
• Farm nearly fully utilised during the day
• youDevise has a very large suite of browser tests