BlogFish

I wrote about Continuous Integration with Ruby on Rails and Subversion before and I finally installed it for a current project. This is indented to be a short HOWTO for it.

CIA, the Continuous Integration Automater, is currently only available through svn. You can get it from http://dev.rubyonrails.com/browser/tools/cia/trunk/.

For those who do not know what CIA is, have a look at the README:

A continuous integration server has a hook into the version control system (Subversion in case of CIA) and when a commit is made it…

1. Find a project that was matched by at least one of the checked in files
2. Checkout that entire project to a temporary directory
3. Run all unit and functional tests on the freshly checked out application
4. Send an email describing the errors if any occurred during the test run

CIA itself is a Rails application that needs to reside on the same machine as the subversion repository. In the CIA application you configure which repositories and which paths of the repositories to use. Test errors will be send by mail and are available in the web-interface.

The following steps will guide you through the setup of CIA.

First checkout CIA to your repository machine. In my case all my web-projects reside under /usr/local/www, so I place CIA there:

# cd /usr/local/www
# svn co http://dev.rubyonrails.org/svn/rails/tools/cia/trunk cia
# cd cia

Now you have to configure CIA like a normal Rails application. CIA needs its own database for managing the projects that should be tested and the build errors.

You can use every supported database like MySQL or PostgreSQL but SQLite seems like a perfect fit for this needs.

The README already has information for creating the CIA-database:

# sqlite db/cia.sqlite # creates and loads the database
sqlite> .read db/sqlite.sql # creates the necessary tables
sqlite> .quit

The default config/database.yml reflects this settings only in production modus. Either change the development environment settings also or make sure that CIA will always use the production settings (like by editing config/environment.rb and set RAILS_ENV = ENV[‘RAILS_ENV’] || ‘production’).

Now configure CIA to use your repository. So start the CIA webapplication. The easiest way is with WEBrick:

# ruby script/server -e production

Point your browser to localhost:3000 and create a new project. The repository has to point to the subversion repository on the server. In my case /home/svn/repos/myproject is a repository created by

# svnadmin create /home/svn/repos/myproject

The path is the branch inside the repository that you want to be checked out and tested like trunk, tags/1.0 or branches/STABLE. Make sure to not use a leading slash.

Now the last thing to do is to configure the subversion post-commit hook so that CIA is called after each commit. Doing so is very easy. Subversion searches the hooks directory inside the repository for scripts that follow a special scheme and calls them with arguments like which repository is used and the revision number.

In my case these scripts are in /home/svn/repos/myproject/hooks:

# ls -laslFh /home/svn/repos/myproject/hooks
2 rw-r-r— 1 jw wheel 2.0K Sep 17 23:10 post-commit.tmpl
2 rw-r-r— 1 jw wheel 1.6K Sep 17 23:10 post-lock.tmpl
4 rw-r-r— 1 jw wheel 2.2K Sep 17 23:10 post-revprop-change.tmpl
2 rw-r-r— 1 jw wheel 1.5K Sep 17 23:10 post-unlock.tmpl
4 rw-r-r— 1 jw wheel 2.9K Sep 17 23:10 pre-commit.tmpl
2 rw-r-r— 1 jw wheel 2.0K Sep 17 23:10 pre-lock.tmpl
4 rw-r-r— 1 jw wheel 2.7K Sep 17 23:10 pre-revprop-change.tmpl
2 rw-r-r— 1 jw wheel 1.9K Sep 17 23:10 pre-unlock.tmpl
4 rw-r-r— 1 jw wheel 2.1K Sep 17 23:10 start-commit.tmpl

The scheme should be obvious. To enable a hook just delete the .tmpl extension or include you own script that uses one of these names.

# mv post-commit.tmpl post-commit

The default post-commit script looks like this:

# grep -v # post-commit
REPOS=”$1”
REV=”$2”

commit-email.pl ”$REPOS” ”$REV” commit-watchers@example.org
log-commit.py—repository ”$REPOS”—revision ”$REV”

Just comment out the last two lines and include this line:

/path/to/cia/script/runner “Agent.build(\”$REPOS\”, $REV)”

In my case it is:

/usr/local/www/cia/script/runner “Agent.build(\”$REPOS\”, $REV)”

This will check out the given revision, repository, and path into the cia/data directory (/usr/local/www/cia/data in my case) as the current commiting user and run all tests.

Beware that the commiting user (the webserver-user if you use mod_dav_svn) must have the permission to write into this directory.

Now every time a new revision gets commited, CIA gets called, checks out the current revision, runs all tests and reports failure or success to the given email addresses.

Well, seems like the Subversion folks wanted to respond to my SVK article by releasing version 1.2 :-)

The main new features of 1.2:

• Optional locking (“reserved checkouts”)
• Full WebDAV autoversioning
• FSFS repository back end is now the default
• Faster access to old revisions
• Many improved APIs
• Many bugfixes

FSFS repository as the new default is a good thing as the old (still fully supported) BerkelyDB had problems when the db was corrupted and always using —fs-type fsfs with the admin tools was annoying. Full WebDAV autoversioning can help with non-technical users.

See the release notes for more information on the changes.

While working on my ruby-gems integration in the FreeBSD ports tree and discovering that it was more complicated that I’d hoped, I got distracted by a very good tutorial to SVK.

SVK is build in Perl around Subversion and Subversion is the successor of CVS. I used Subversion for my recent projects and heard about SVK before but never got the chance to see what’s the difference. SVK adds some nice features like star-merging, lightweight checkout copy management (no .svn or CVS directories), changeset signing and verification to Subversion while being faster and smaller in space.

These are nice good-to-have’s but the real power of SVK is its ability to mirror remote Subversion repositories. Actually it also can mirror CVS and perforce repositories, but I will concentrate on Subversion.

This means that a developer can mirror the main repository then disconnect this PC and work offline while using source control features like branches, tags, commits and backouts on his local computer. Eventually he will reconnect to the network, sync his mirror, merge the changes to his local copy/working directory. He can continue to work on his local version until he wants to commit the changes to the “real”, remote repository.

This is very useful if you do not always have a network connection to the main repository or when you want to develop independently of the main repository while still using source control and still sync with the main server. The second case is useful if you do not have a commit bit and locally develop a patch.

Hopefully the advantage is clear. you can get a glimpse on the workflow here.

I will give a very short intro, please read the mentioned tutorial for details.

After installing SVK (cd /usr/ports/devel/svk && make install clean on FreeBSD, fink install svk on OS X), I initialize the depot (repo in svn):

svk depotmap—init

Then I mirror and sync the remote Subversion repository:

svk mirror http://svn.example.com/project/trunk //project/trunk
svk sync //project/trunk

//project/trunk is the local repository that you can freely choose. Now I create a branch for local editing. If I wouldn’t do that and just checkout the trunk to a working copy, SVK will sync every commit I make to the local repository to the remote one. Often this is not what was intented. so create a local branch:

svk cp -m ‘local branch’ //project/trunk //project/local

Now I can checkout a working copy and work like I used with Subversion:

cd /tmp/work
svk co //project/local project
vim project/somefile
vim project/another
svk commit
...

Say in the meanwhile the project evolved and I want to sync with it in order to get the latest features and code:

svk sync //project/trunk

The local repository is now synced but my branch isn’t. So I merge the changes to my branch. This can get dirty with Subversion after several merges as you have to explicitly name the revision since you last merged from the trunk. With SVK and it’s star-merge this is easy:

svk smerge //project/trunk //project/local

The only thing left to do is merging the branch with the working copy through a svk up if you have one. Otherwise were officially synced!

To sync the repositories in the other way round, that means to commit your local changes to the remote server, just change the repositories-arguments to svk smerge.

So we merge our branch with our local trunk:

svk smerge //project/local //project/trunk

and we’re done as SVK syncs the local repository automatically with the remote one (in both ways).

This was only a small introduction, look for the svk push and pull commands to ease your work with SVK. But using the smerge command you have a better understanding of what is really going on.

I wrote about CIA before. It is a hook to subversion that checks out the tree after every commit and runs all tests.

Patrick Lenz published a short article about how to setup CIA. Like everything in Rails it is very easy.

UPDATE:
I wrote a small HOWTO on CIA, rails, and subversion here.

As a follower of Extreme Programming and Test Driven Development I strive for Continuous Integration. This means that after every commit an agent automatically builds the project, runs every test and bail in case of errors. This should insure that the project is always build-able and that no errors are sneaking in. The automation is very important as developers tend to be sloppy and forget to build or run all tests. Also if the complete build is taking too long in order to test it locally after each change, Continuous Integration can help. Have a look at this article by Martin Fowler for more about Continuous Integration and why you should use it.

While poking around in the Ruby on Rails subversion source code tree I found CIA, the Continuous Integration Automater. Seems like again Ruby on Rails is a leader in integrating known processes, patterns and tools into an excellent framework for web development. CIA follows the above described workflow of Continuous Integration and is easy to set up. It works as a post-commit hook in Subversion. Quoting the README:

Setting up CIA requires that you’re on the same machine as the repository. If
you haven’t already activated the post-commit hook, then copy /path/to/repos/hooks/post-commit.tmpl
to /path/to/repos/hooks/post-commit and add to the bottom of it:

/path/to/cia/script/runner “Agent.build(”$REPOS”, $REV)”

Another usefull tool developed in the Ruby community is DamageControl. DamageControl comes as a ruby-gem and can build several dependent projects. The project can be in an arbitrary language as long as it can be built by executing a simple command line. That means that if your project uses for example Ant, Make, Rake or Rant you can use DamageControl. Refer to the documentation for more information.

I will start using CIA for my next Ruby on Rails project and will publish my experiences with this tool.

UPDATE:
I wrote a small HOWTO on CIA, rails, and subversion here.