Problems Adopting CI

From CitconWiki
Jump to navigationJump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

Problem: End-of-day checkins breaking the build or production

  • Background/Poll of Attendees:
    • Some teams have very short CI windows (5 minutes), others doing nightlies
    • Very few doing promoted/sequential builds/graduated pipelines
  • Use local Jenkins instances in VMs that each dev can use to test their work before committing
  • Favoring small, frequent commits over infrequent, big commits; needs education of devs to avoid siloing and holding on to code
  • Use flaggable features to incrementally promote parts of a larger collection of features
  • Mitigation proxies: route some small subset of traffic to new version/feature. Incrementally increase traffic over time until a problem arises or 100% of traffic is reached.

Question: What to do when CI can't keep up?

  • Large amounts of customer data that doesn't match production
  • Need to manage data in the same way as other configuration features

How to deal with configuration drift?

  • e.g., tools/libraries/etc only on CI server or different from devs' version
  • Configuration has to be the same!
    • Build process could include check for needed configuration and installs it if not present
    • Use VMs with exact needed configuration; create/destroy per build to guarantee no stale artifacts, etc.

How to maintain multiple software versions?

  • Archive past versions to maintenance branches
  • Bug fixes pushed to all branches and propagated up
  • Realize that build team's customer is dev ops; know who your customer is.
  • Customers are distinct from stakeholders

How to deal with limited rights on production servers?

  • Understand that ops' primary fears are configuration drift & security
  • Logs can be exported to a place accessible to devs/testers
  • Oppositional culture (devs vs ops, testers vs devs) can play a role; avoid us vs. them thinking!
  • Need to find ways of getting what you need without access; devs need to work with ops to get what is needed