Large Scale CI

From CitconWiki
Jump to navigationJump to search

EA System

  • 4 years to develop!
  • end to end automated pipeline
  • lots of investment
  • System components
    • Perforce (games use lots of binary data, Perforce very good for that type of data)
    • Automation using buildforge
      • not very good tracking of configuration changes in buildforge
      • CruiseControl actually was very good for tracking build changes in version control
      • Before buildforge, clean build took 8-10 hours
      • buildforge very critical. No one able to make changes without pair programming
    • 500 blades, fully virtualised, running vmware esx
      • For every build, a new VM was spun up with the required software and configuration, and then shut down when the build finished
      • Incredibuild, spun up buildforge instance
      • Nothing special about each vm image
    • netapp san storage
      • netapp snapshotting (copy on write) was very good at optimising disk space. e.g. 2 different files that were mostly the same, only the diffences would be stored
    • parametrised projects
    • Juice: platform testing scripting language (propriatory scripting engine for games)
      • No testing of "correctness" of AI, mainly worried about the fact that the game didn't crash
      • Correctness mainly limited to menu interations
    • Themis
      • Allows juice scripts to be mapped to source code changes
      • After a checkin, themis would suggest the best juice scripts to run to get the best possible coverage of code
      • Allows analysis of QA coverage versus required coverage
      • Reduced manual testing by huge amount
    • Replay technologies
      • Able to record the physical state of a piece of hardware, and replay it later for testing. Replay it on hardware with a debugger attached
  • Biggest challenges
    • Getting money to invest!
    • Wasn't obvious to management that investing in the hardware would save money
    • Management now very happy that the investment was absolutely worth it

youDevise CI Farm

  • Much smaller than EA!
    • Around 20 Hudson build slaves, and a similar number of slaves for Selenium Grid
  • Complex configuration, needs to be simplified
  • Farm nearly fully utilised during the day
  • youDevise has a very large suite of browser tests