Reproducible CI

Reproducible CI

Can I do this with Windows servers? Yes.

Versioning CI tools by having a job that commits the CI configuration to source control every hour. Using config history plugin for Jenkins gives you history of config, but only by using the plugin.

Has someone upgraded the compiler or patched the OS? Has that impacted the build?

Some clients spin up a VM from an image and set the clock to a known time, leaving nothing to chance.

What is the end goal of reproducible CI?

Being able to bring back the CI environment if it comes away.

Some businesses need to be able to reproduce the application years in the past. If you're seeing weird behavior with your application, it may be caused by the CI server, and this way you can see what was happening with the build server for a specific artifact. Some clients will want to stay on the same version for years and years, so you may need to recreate a build that hasn't been run in years. With large and/or remote teams, being able to see who made what change and why is critical, especially if something goes wrong. With CI/versioning, you can promote changes to your pipeline.

If you can have a reproducible build system, developers can prove to themselves that what they are going to add won't break the build. No "it works on my machine" excuses.

Tools:

• Vagrat
• Docker

Is there a need for a "meta-pipeline" (pipeline for the pipeline) that builds and promotes the build system? Build systems are products, consumed by developers, and testing needs to happen before the change is pushed to prod server.

We don't hold the pipeline to the same processes we hold for the source code we're building.

Is the issue that the team is too big? Do we want our team of ten managing a build server, or do we want them delivering value to customers? Do I want 100 teams with their own build server, or do a want a couple guys managing a build server for 1,000 developers? Having the small team manage their build system empowers them about the build and deployment process. Having a single team manage it for a huge group of developers gets the developers out of touch with the deployment process. Developers care more about getting stuff done over reproducibility -- they'll install something to "get it to work" and then forget how it got there.

• Developers manage their server, but the slaves are rebuild nightly.
• All teams have their own build server, but agree on a specific deliverable type and publish it to an artifact repository. There's a master build server that watches the repo and does the integration of all the components.

Does anyone care about traceability? Getting the code is easy. What about the tests that run? How was the build server set up that day?

We can reproduce what we built, can we reproduce how we deployed it? If we change all the build scripts? If we change what tool we used?

Don’t do * includes if you can avoid it.

Tool chain from checkin -> deployment to production

SCM git svn hg ClearCase Perforce Build Server “Runs the build” Transform source + dependencies into an artifact. (May also run tests.) Compile, link, package, test Static Analysis Jenkins BuildForge AHP CruiseControl Artifact repo Nexus Artifactory S3 SCM Leave it in the build tool Push it to deployment tool Infrastructure Would follow the same chain to go from “source” to “image.” Long lived physical environments that get set up once Virtual machines that are treated like long-lived physical machines. Stand up virtual instances per build A person might build the machine, build the golden image, build a custom script. Deploy Chef/Puppet Capistrano uDeploy Orc Deploy Shell/Perl/Python/Ruby/etc scripts A person does everything manually following a document Database Liquibase Rails Migrations Dacpac Play Evolution A person that updates the database. Test Performance Integration Unit Deployment/Smoke Functional Load Security Installation/Manifest Check (make sure all expected files exist) A person does everything manually following a document Bees with Machine Guns HP LoadRunner/WinRunner/etc Cucumber Selenium Fitnesse Spock UAT / Business Signoff Blue / Green deployments Canary deployment Person clicks a button Manual signing of document Deploy to Production? Same as deploy to non-production A person does a partial deployment

Should a build server deploy? Yes Get it out there so people can use it No Slow tests aren’t run Deploy to one environment or many?

Conclusion: script all the things. Version all the things. You can do it on Windows. Treat the systems the same way you are (should be?) treating the software going to the users. It’s worth it for any size team: one person, ten people, a thousand people.