Difference between revisions of "Problems Adopting CI"

From CitconWiki
Jump to navigationJump to search
(Initial dump of notes from session)
 
m (Formatting fixes)
 
Line 1: Line 1:
Problem: End-of-day checkins breaking the build
+
Problem: End-of-day checkins breaking the build or production
- Some teams have very short CI windows (5 minutes), others doing nightlies
+
* Background/Poll of Attendees:
- Very few doing promoted/sequential builds/graduated pipelines
+
** Some teams have very short CI windows (5 minutes), others doing nightlies
- Local Jenkins instances in VMs that each dev can use
+
** Very few doing promoted/sequential builds/graduated pipelines
- Favoring small, frequent commits over infrequent, big commits; needs education of devs to avoid siloing and holding on to code
+
* Use local Jenkins instances in VMs that each dev can use to test their work before committing
- Use flaggable features to promote parts of a larger goal
+
* Favoring small, frequent commits over infrequent, big commits; needs education of devs to avoid siloing and holding on to code
- Mitigation proxies: route some small subset of traffic to new version/feature. Incrementally increase traffic over time until a problem arises or 100% of traffic is reached.
+
* Use flaggable features to incrementally promote parts of a larger collection of features
 +
* Mitigation proxies: route some small subset of traffic to new version/feature. Incrementally increase traffic over time until a problem arises or 100% of traffic is reached.
  
 
Question: What to do when CI can't keep up?
 
Question: What to do when CI can't keep up?
- Large amounts of customer data that doesn't match production
+
* Large amounts of customer data that doesn't match production
- Need to manage data in the same way as other configuration features
+
* Need to manage data in the same way as other configuration features
  
 
How to deal with configuration drift?
 
How to deal with configuration drift?
- e.g., tools/libraries/etc only on CI server or different from devs' version
+
* e.g., tools/libraries/etc only on CI server or different from devs' version
- Configuration has to be the same!
+
* Configuration has to be the same!
-- Build process could include check for needed configuration and installs it if not present  
+
** Build process could include check for needed configuration and installs it if not present  
-- Use VMs with exact needed configuration; create/destroy per build to guarantee no stale artifacts, etc.  
+
** Use VMs with exact needed configuration; create/destroy per build to guarantee no stale artifacts, etc.  
  
How to maintain multiple software versions?
+
How to maintain multiple software versions?
- Archive past versions to maintenance branches
+
* Archive past versions to maintenance branches
- Bug fixes pushed to all branches and propagated up
+
* Bug fixes pushed to all branches and propagated up
- Realize that build team's customer is dev ops; know who your customer is.
+
* Realize that build team's customer is dev ops; know who your customer is.
- Customers are distinct from stakeholders
+
* Customers are distinct from stakeholders
  
 
How to deal with limited rights on production servers?
 
How to deal with limited rights on production servers?
- Understand that ops' primary fears are configuration drift & security
+
* Understand that ops' primary fears are configuration drift & security
- Logs can be exported to a place accessible to devs/testers
+
* Logs can be exported to a place accessible to devs/testers
- Oppositional culture (devs vs ops, testers vs devs) can play a role; avoid us vs. them thinking!
+
* Oppositional culture (devs vs ops, testers vs devs) can play a role; avoid us vs. them thinking!
- Need to find ways of getting what you need without access; devs need to work with ops to get what is needed
+
* Need to find ways of getting what you need without access; devs need to work with ops to get what is needed

Latest revision as of 08:23, 4 October 2015

Problem: End-of-day checkins breaking the build or production

  • Background/Poll of Attendees:
    • Some teams have very short CI windows (5 minutes), others doing nightlies
    • Very few doing promoted/sequential builds/graduated pipelines
  • Use local Jenkins instances in VMs that each dev can use to test their work before committing
  • Favoring small, frequent commits over infrequent, big commits; needs education of devs to avoid siloing and holding on to code
  • Use flaggable features to incrementally promote parts of a larger collection of features
  • Mitigation proxies: route some small subset of traffic to new version/feature. Incrementally increase traffic over time until a problem arises or 100% of traffic is reached.

Question: What to do when CI can't keep up?

  • Large amounts of customer data that doesn't match production
  • Need to manage data in the same way as other configuration features

How to deal with configuration drift?

  • e.g., tools/libraries/etc only on CI server or different from devs' version
  • Configuration has to be the same!
    • Build process could include check for needed configuration and installs it if not present
    • Use VMs with exact needed configuration; create/destroy per build to guarantee no stale artifacts, etc.

How to maintain multiple software versions?

  • Archive past versions to maintenance branches
  • Bug fixes pushed to all branches and propagated up
  • Realize that build team's customer is dev ops; know who your customer is.
  • Customers are distinct from stakeholders

How to deal with limited rights on production servers?

  • Understand that ops' primary fears are configuration drift & security
  • Logs can be exported to a place accessible to devs/testers
  • Oppositional culture (devs vs ops, testers vs devs) can play a role; avoid us vs. them thinking!
  • Need to find ways of getting what you need without access; devs need to work with ops to get what is needed