Difference between revisions of "Problems Adopting CI"
From CitconWiki
Jump to navigationJump to search (Initial dump of notes from session) |
m (Formatting fixes) |
||
Line 1: | Line 1: | ||
− | Problem: End-of-day checkins breaking the build | + | Problem: End-of-day checkins breaking the build or production |
− | + | * Background/Poll of Attendees: | |
− | + | ** Some teams have very short CI windows (5 minutes), others doing nightlies | |
− | + | ** Very few doing promoted/sequential builds/graduated pipelines | |
− | + | * Use local Jenkins instances in VMs that each dev can use to test their work before committing | |
− | + | * Favoring small, frequent commits over infrequent, big commits; needs education of devs to avoid siloing and holding on to code | |
− | + | * Use flaggable features to incrementally promote parts of a larger collection of features | |
+ | * Mitigation proxies: route some small subset of traffic to new version/feature. Incrementally increase traffic over time until a problem arises or 100% of traffic is reached. | ||
Question: What to do when CI can't keep up? | Question: What to do when CI can't keep up? | ||
− | + | * Large amounts of customer data that doesn't match production | |
− | + | * Need to manage data in the same way as other configuration features | |
How to deal with configuration drift? | How to deal with configuration drift? | ||
− | + | * e.g., tools/libraries/etc only on CI server or different from devs' version | |
− | + | * Configuration has to be the same! | |
− | + | ** Build process could include check for needed configuration and installs it if not present | |
− | + | ** Use VMs with exact needed configuration; create/destroy per build to guarantee no stale artifacts, etc. | |
− | + | How to maintain multiple software versions? | |
− | + | * Archive past versions to maintenance branches | |
− | + | * Bug fixes pushed to all branches and propagated up | |
− | + | * Realize that build team's customer is dev ops; know who your customer is. | |
− | + | * Customers are distinct from stakeholders | |
How to deal with limited rights on production servers? | How to deal with limited rights on production servers? | ||
− | + | * Understand that ops' primary fears are configuration drift & security | |
− | + | * Logs can be exported to a place accessible to devs/testers | |
− | + | * Oppositional culture (devs vs ops, testers vs devs) can play a role; avoid us vs. them thinking! | |
− | + | * Need to find ways of getting what you need without access; devs need to work with ops to get what is needed |
Latest revision as of 08:23, 4 October 2015
Problem: End-of-day checkins breaking the build or production
- Background/Poll of Attendees:
- Some teams have very short CI windows (5 minutes), others doing nightlies
- Very few doing promoted/sequential builds/graduated pipelines
- Use local Jenkins instances in VMs that each dev can use to test their work before committing
- Favoring small, frequent commits over infrequent, big commits; needs education of devs to avoid siloing and holding on to code
- Use flaggable features to incrementally promote parts of a larger collection of features
- Mitigation proxies: route some small subset of traffic to new version/feature. Incrementally increase traffic over time until a problem arises or 100% of traffic is reached.
Question: What to do when CI can't keep up?
- Large amounts of customer data that doesn't match production
- Need to manage data in the same way as other configuration features
How to deal with configuration drift?
- e.g., tools/libraries/etc only on CI server or different from devs' version
- Configuration has to be the same!
- Build process could include check for needed configuration and installs it if not present
- Use VMs with exact needed configuration; create/destroy per build to guarantee no stale artifacts, etc.
How to maintain multiple software versions?
- Archive past versions to maintenance branches
- Bug fixes pushed to all branches and propagated up
- Realize that build team's customer is dev ops; know who your customer is.
- Customers are distinct from stakeholders
How to deal with limited rights on production servers?
- Understand that ops' primary fears are configuration drift & security
- Logs can be exported to a place accessible to devs/testers
- Oppositional culture (devs vs ops, testers vs devs) can play a role; avoid us vs. them thinking!
- Need to find ways of getting what you need without access; devs need to work with ops to get what is needed