It’s Okay to Break the Build
“My name is Stuart Halloway, and I believe it is OK to break the build.”
Several developers have recently asked me “How should developers who break the build be punished?” With only that information, my answer is easy: They shouldn’t.
Of course breaking the build is not a good thing, but let me make a couple of hypothetical examples to demonstrate the point.
Example 1. Jane’s Birdhouses (JB)
JB is an ecommerce + social networking site for a brick-and-mortar birdhouse store. Built with Test-Driven Development (TDD), JB has three test suites:
- a few hundred unit tests that execute in about 20 seconds.
- a few dozen integration tests that execute in about 90 seconds.
- a few dozen acceptance tests in Cucumber and Selenium that must be run multiple times against different browser versions. As a result, these “long tests” take about 30 minutes to execute.
Now, let’s consider how the JB build might break due to developer negligence:
- A developer could commit code without bothering to run the unit tests or integration tests locally.
- A developer could commit code knowing that the tests were broken.
Both of these things very bad behavior, and should be condemned by any agile team.
On rare occasions, extenuating circumstances might justify such “negligent” commits, but documenting such corner cases is not my purpose here. More interesting to me are the legitimate, respectable commits that might nevertheless break the build.
For example, what if a developer commits code to JB that passes the unit and integration tests, but breaks the long-running tests? That’s not good, but it may be better than the alternative. If you insist that developers run the “long tests” locally before committing to the build box, then you cannot call yourself agile. Rapid feedback is critical to agility, and taking several 30-minute breaks a day will never meet that standard.
So you should not insist on perfect builds. What should you do if a commit breaks the long tests? Stop what you are doing, figure out why, and fix it. The problem will fall into one of three categories:
- The problem could have been detected by a fast test, and that fast test was an obvious thing to write. Solution: write the fast test, and beat up the negligent dev.
- The problem could have been detected by a fast test, but the needed long test was not obvious until the long test found it. No negligence here. Solution: write the fast test.
- The problem could not have been detected by a fast test, or the cost of writing the fast test is extremely high. This case sucks—but it does justify running the long tests. Fix the problem, and know that similar problems in the future will also break the build.
Another possibly legitimate build-breaker is uncovering a hitherto-unknown dependency in an application’s setup. Similar rules apply:
- If the problem should have been obvious, then shame on the developer. Example: adding a necessary Rubygem on a local box without bothering to add it to the build (ie config.gems in a Rails project)
- If the problem was more subtle, then just fix, learn, and move on. Example: the latest version of support library Foo introduces a new dependency on Bar. The authors of Foo did not document the dependency clearly, and the dev didn’t notice because he already had Bar installed.
Example 2. Jim’s Enterprise App (JEA)
JEA is a much bigger, more serious application than JB. It accesses five separate databases, corresponding to five separate business units. The tests are correspondingly more complex, and the test suite takes four hours to run.
The JEA team is committed to quality. The original application didn’t have unit tests, but the team has worked heroically to add them, and code coverage now stands at 80.1%. Developers are docked on their performance evaluations if they commit code that
- breaks a test
- drops coverage below 80%
How should we interpret this? Despite possible good intentions, the JEA team fails agile at Square 1. First, the developers have to run a four-hour test suite locally before committing. Otherwise, they risk automatic censure for breaking the build.
Second, the coverage ain’t going to go much above 80%, ever. Think about it: Bill writes a new module. When he runs the tests, coverage goes up to 80.2%, but a test somewhere else in the app breaks. If Bill simply deletes the other test, he meets both his objective performance criteria: no broken tests, and coverage stays above 80%. Talk about perverse incentives!
In my experience auditing code, I have seen far more codebases like JEA than like JB. My guess is that most projects never partition their tests into slow and fast, even when the test suite has grown so slow that agile is a dimly-remembered dream. Moreover, should JEA even be a single project? Probably not. Statistically, large projects suck, so if you have any reasonable way to break your project into little ones, you probably should.
Wrapping Up
Is breaking a build bad? Yes, sometimes. Usually, even. But context matters, and teams that ignore context will never be agile.
