Problems Adopting CI

From CitconWiki
Revision as of 08:11, 4 October 2015 by Pkerchen (talk | contribs) (Initial dump of notes from session)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigationJump to search

Problem: End-of-day checkins breaking the build - Some teams have very short CI windows (5 minutes), others doing nightlies - Very few doing promoted/sequential builds/graduated pipelines - Local Jenkins instances in VMs that each dev can use - Favoring small, frequent commits over infrequent, big commits; needs education of devs to avoid siloing and holding on to code - Use flaggable features to promote parts of a larger goal - Mitigation proxies: route some small subset of traffic to new version/feature. Incrementally increase traffic over time until a problem arises or 100% of traffic is reached.

Question: What to do when CI can't keep up?

- Large amounts of customer data that doesn't match production
- Need to manage data in the same way as other configuration features

How to deal with configuration drift?

- e.g., tools/libraries/etc only on CI server or different from devs' version
- Configuration has to be the same!
-- Build process could include check for needed configuration and installs it if not present 
-- Use VMs with exact needed configuration; create/destroy per build to guarantee no stale artifacts, etc. 
How to maintain multiple software versions?
- Archive past versions to maintenance branches
- Bug fixes pushed to all branches and propagated up
- Realize that build team's customer is dev ops; know who your customer is.
- Customers are distinct from stakeholders

How to deal with limited rights on production servers?

- Understand that ops' primary fears are configuration drift & security
- Logs can be exported to a place accessible to devs/testers
- Oppositional culture (devs vs ops, testers vs devs) can play a role; avoid us vs. them thinking!
- Need to find ways of getting what you need without access; devs need to work with ops to get what is needed