DistributedSCMInTheCorporateWorld

Distributed SCM In the Corporate World

Motivation

The motivation of this session was to find out how people are using Distributed SCM/Version Control (e.g. git/Mercurial/Bazaar) in a corporate setting. These SCMs have caught on quickly in open source development, but few development projects work like the Linux kernel. I was interested in seeing if and/or how people were using the new capabilities of these SCMs in their own projects.

Introduction

As many of the attendees were not too familiar with distributed version control we started with a very high level overview of how it works. The traditional SCM model involves a central SCM server which holds the repository - both the file data and all the history. Developers "check out" working copies of the files on their client machines, make changes and "commit" them back to the central server. They obtain the commits made by other developers by "updating" from the central server. There is a fundamental difference between the central server (repository) and client machines (working copies).

In a distributed model, every developer has a fully-fledged repository on their own machine. This includes both the file data and the history. Developers can make changes and "commit" them to their own, local repository. Commits are shared between repositories by "pushing" or "pulling" them. There is no requirement for a central server at all, although in practice there will usually be a machine (or machines) designated to a similar task. Developers will push changes to such a machine to make them public (i.e. when they would have committed in the centralised model), and pull from that machine to obtain other's commits (i.e. when they would have updated in the centralised model).

There are a lot more details to understand, but we didn't get too far into it - preferring instead to move on to some real-life experiences.

Practical Experience

At this point we invited those that had some real-world experience to talk a bit about how they use distributed SCM. It turns out we had a few attendees that were actively using git, but none that were using alternatives.

Topology

I asked our git users if they used a topology resembling the centralised model, or something that took advantage of the distributed nature of the SCMs. It turns out they were all using a defacto centralised model: one machine was designated as the "central" git repository. This machine was used by all developers to integrate their changes (pushing their own changes and pulling those of others). The repository was the "one true source" used to run continuous integration and build releases. (Actually one of the teams didn't use a separate CI server, instead preferring to rely on developers to test and integrate frequently as they worked.)

One team had planned to use a more complex topology to help cope with network issues. In this case the team was split into two remote locations, and were uncertain of the reliability of the network connection between them. Their original setup involved an "intermediate" integration repository at each location, each connected to a single central repository. Developers in each location would push to their local intermediate frequently, and changes would then be pushed from the intermediate to the central server. If the network connection between the teams was broken, they would always at least be able to integrate locally using their intermediate server. In reality the network performed well enough in practice that the team just scrapped the intermediate servers.

Workflow

We moved on to discuss how using git influenced the developer workflow. Two major points came out of this, as described below.

Local Commits

The git users used and appreciated the ability to make local commits. They would checkpoint their changes frequently in their local repository - even if they were half-baked. This allowed them to explore different angles whilst still being able to backtrack. The question of whether this made developers integrate with each other less often was raised, although nobody reported this happening in practice. It seems that if the team have the "CI attitude" this shouldn't be an issue. We touched on the ability to rewrite your local history so that you can push simpler, logical commits to integration repositories (e.g. git rebase).

Interestingly someone mentioned a commercial central SCM (IIRC Serena Dimensions) that allowed you to stage commits in a similar way - you could commit to the central repository with yet sharing with other users. It was also mentioned that this affect can be achieved via branching - although it tends to complicate your workflow more.

Strong Merging

The git users also appreciated its strong support for merging. Some of the popular central SCMs (Subversion, we're looking at you) have been historically weak in this area. It was mentioned that merge tracking exists in recent Subversion versions, and has existed in other central SCMs for a long time. Still, practical experience suggests that the merging algorithm in git is one of the better ones. One git user explained a pain point involving merge "races" between developers - to be honest the exact details escape me. It involved overlapping conflicts in a single file, and git asking a developer to re-merge conflicts that had already been resolved. I suspect this is a relatively rare case. An interesting tangent from this that we didn't have time to explore was the tension between refactoring and merging.

CI

We also discussed the interaction between distributed SCMs and CI. As our git teams used a centralised topology, their CI setup was no different from that used with traditional SCMs. The obvious place for the CI server is talking to the central git repository.

We also discussed the ability to get "personal builds" (also called "pre-flight" -- CI builds of your new changes before you share them with your team) without requiring special CI server support. Instead you can set up a project in your CI server to monitor your local git repository. When you make local commits they can be picked up and tested - and you can hold back on pushing them until you get the green light. None of the attendees seemed to be using this model, but I know some Pulse customers that use git this way. (In fact some of their developers have their own local "staging" git repostories for this purposes - showing the flexibility of everything being a repository.)

Conclusions

I took a few main points away from this session:

1. Of the distributed SCMs, there is not much traction in the corporate world just yet, although git appears to have gained a foothold. (Obviously our sample size is small, but I also expect CITCON attendees to be closer to the edge than the average team.)
2. Where distributed SCMs are used, the topology is still like the centralised model. However, the ability to easily clone and move changes between repositories presents opportunities to work around issues like painful networks (contrast this to special proxy servers which are needed in similar scenarios with centralised SCMs).
3. The people using git liked it primarily for its more flexible workflow and better merging. It's conceivable to have this in the centralised model too, but no single centralised contender was mentioned.
4. So far the use of distributed SCMs didn't seem to have practical implications for CI - probably due to the use of a centralised topology.

I'm still keen to talk to anyone who has tried or is using a different topology with a distributed SCM. Anyone?