Testing against multiple Rails versions
It should be obvious from the fact that we built RunCodeRun that we at Relevance believe strongly in continuous integration. CI is more than just testing—it gives you something qualitatively different, over and above what good testing discipline gives you.
One example of that is what Chris Eppstein (one of the Haml and Sass team) did last week. He’s using RunCodeRun to test the Haml gem and plugin against multiple versions of Rails (including the 2.3 release candidate). Chris’ latest Haml build status page shows the full test suite being run four times before reporting success.
When we run your project’s tests under RunCodeRun, we set the RUN_CODE_RUN environment variable, so your tests can know where they’re running and do something special, if desired. Chris makes use of that in the first part of his Rakefile to set a different default task for RunCodeRun builds:
if ENV["RUN_CODE_RUN"] == "true"
task :default => :"test:rails_compatibility"
else
task :default => :test
end
The test:rails_compatibility task runs the entire test suite against several Rails versions, failing if the tests fail against any of those releases:
namespace :test do
desc "Test all supported versions of rails. This takes a while."
task :rails_compatibility do
`rm -rf test/rails`
puts "Checking out rails. Please wait."
`git clone git://github.com/rails/rails.git test/rails` rescue nil
begin
rails_versions.each do |version|
Dir.chdir "test/rails" do
`git checkout #{version}`
end
puts "Testing Rails #{version}"
Rake::Task['test'].reenable
Rake::Task['test'].execute
end
ensure
`rm -rf test/rails`
end
end
end
We’re absolutely delighted; RunCodeRun is perfect for this. As Chris’ comment points out, it’s a time-consuming process, so you don’t want to run all of these tests before every checkin; that would slow your development down. But you do want to learn relatively quickly if you’ve done something that will fail on older Rails versions. It’s an ideal workflow: run the ordinary test task before each checkin, and then have RunCodeRun test against all of the supported Rails versions after each checkin.
We hope to make this even easier in the future … we’ve already got a few ideas. For now, though, Chris’ approach works just fine.
We hope other plugin developers will follow Chris’ example. This makes it easier to support older Rails versions. By testing against the upcoming release candidate, you can ensure that you’ll be able to support the new version right out of the box. By testing against edge, you can get early warning of things that might break in the future.
