Continuous Integration Server in a box

So at work, we needed a simple continuous integration server for testing purposes. The lab I work in is closed off from internet access and I don't have admin privileges, so I needed a solution that would work with no help from IT. Using git as my configuration management tool, here is what I ended up using, rather nicely as it turns out.

The product is an open-sourced tool written in node.js called Concrete [1]. The setup is basically this:

On a machine with internet access,

1. Download and drop on the computer node.exe [2]. As an example, use c:\Users\keith\tmp\node

2. Download and expand npm archive [3] in the same directory as where node.exe resides.

3. Open a command prompt and go to the node.exe directory Run npm install concrete

  C:\Users\keith\tmp\node\npm install concrete

4. (Optional) Install node's IRC module. The node IRC module will be used to modify Concrete to add IRC notification and demonstrate how easy it is to hack this simple tool.

  C:\Users\keith\tmp\node\npm install irc

5. Get unxutils [4] or gow [5] or some form of curl.exe. Curl is used via git-hooks to signal the continuous integration server that a change has been pushed up to the repository.

6. Zip up the node/npm/npm_modules stuff

7. Download mongodb [6]

8. (Optional) Get bewareircd [7] and irrsi [8], an IRC server and client.

9. Transfer the node stuff and mongodb archives as well as curl and the optional downloads to a machine on the target network.

Now on your desired test machine.

1. Unzip the node stuff. I will use C:\tools\concrete_in_a_box\node as an example

  C:\tools\concrete_in_a_box\node\node.exe
  C:\tools\concrete_in_a_box\node\npm.cmd
  C:\tools\concrete_in_a_box\node\node_modules\
  etc.

2. Unzip mongodb. Again, I will use C:\tools\concrete_in_a_box as an example

  C:\tools\concrete_in_a_box\mongo

3. Open a command prompt and change paths to the mongo directory. Start mongo, making sure that the invocation states where the database will reside. The default is C:\data\db, but I prefer to keep everything self-contained. So I make a directory inside the concrete_in_a_box directory called mongo_data.

  C:\tools\concrete_in_a_box\mongo\mongod.exe --dbpath ..\mongo_data

4. Open another command prompt. Clone the git repository of interest. For this simple case, let's assume that there is repository of interest in C:\repos\my_project and we clone it in the concrete_in_a_box directory

  C:\tools\concrete_in_a_box> git clone c:\repos\my_project 

5. Add a concrete job runner. As an example, assume that there is a build script batch file named build.bat as part of the project

  C:\tools\concrete_in_a_box\my_project> git config --add concrete.runner="build.bat"

build.bat might be something as simple as invoking msbuild to build a Visual Studio solution. As an example,

@REM build.bat
@echo off
@REM make sure that msbuild is available (assuming VS2012)
@if "%VSINSTALLDIR%"=="" call "C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Tools\vsvars.bat"
msbuild my_project.sln /p:Configuration=Debug;VisualStudioVersion=11.0
@exit \b %ERRORLEVEL%

6. Use hooks to signal to Concrete that changes have occurred. In this particular example, we modify the .git/hooks/post-receive hook of the repository we cloned for Concrete to invoke curl to post just blank data to the IP and port that Concrete will be running on.

  C:\repos\my_project\.git\hooks> <edit post-receive>

#!/bin/sh
#
# An example hook script for the "post-receive" event.
#
# The "post-receive" script is run after receive-pack has accepted a pack
# and the repository has been updated.  It is passed arguments in through
# stdin in the form
#  <oldrev> <newrev> <refname>
# For example:
#  aa453216d1b3e49e7f6f98441fa56946ddcd6a20 68f7abf4e6f922807889f52bc043ecd31b79f814 refs/heads/master
#
curl localhost:4567 -d Build

7. Open another command prompt. Add to the execution path the node.exe directory and the node_modules\.bin directory

  C:\> set path=%PATH%;C:\tools\concrete_in_a_box\node;C:\tools\concrete_in_a_box\node\node_modules\.bin

and then execute concrete

  C:\tools\concrete_in_a_box\my_project> concrete .

8. Open your web browser and observe Concrete sugary goodness at http://localhost:4567.
Now, anytime that a change is commited and pushed to the repository at c:\repos\my_project, it triggers the post-receive hook which issues a POST command to the Concrete webserver, which invokes the runner. In this particular case, it pulls down the latest from its git origin (c:\repos\my_project) and then runs build.bat.

9. If you have not created a build-worked or build-failed hook file in your .git\hooks directory, it will bark at you saying it cannot find it. I simply created little bat files that echo whether the build succeeded or failed. One thing I did was I modified the Concrete git.js library file and explicitly called the files it was looking for to be build-worked.bat and build-failed.bat.

diff git.js original_git.js

23,24c23,24
<       git.failure = path.normalize(target + '/.git/hooks/build-failed.bat');
<       git.success = path.normalize(target + '/.git/hooks/build-worked.bat');
---
>       git.failure = path.normalize(target + '/.git/hooks/build-failed');
>       git.success = path.normalize(target + '/.git/hooks/build-worked'); 

Optional fun!
So we also included some IRC stuff; the lab we have does not have an email server set up and I wanted to have instantaneous feedback on when a build occurred and what the outcome was. I figured it would be very simple to plumb in a little IRC bot, and using some example code from here [9], I was able to get one up and  running in about 15 minutes, including the setting up of the IRC server and client. The only modification I did was to runner.js

diff runner.js original_runner.js

10,25d9
<
<   // Create the configuration
<   var ircConfig = {
<       channels: ["#concrete" ],
<       server: "my.ownircserver.net",
<       botName: "concretebot"
<   };
<   // Get the lib
<   var irc = require("irc");
<
<   // Create the bot name
<   var ircBot = new irc.Client(ircConfig.server, ircConfig.botName, {
<       channels: ircConfig.channels
<   });
<
<
114d97
<                 ircBot.say(channel, "Build failed!");
121d103
<                 ircBot.say(channel, "Build succeeded!"); 

Unzip the bewareircd archive and modify ircd.conf so that the server name matches the server name in runner.js (e.g., my.ownircserver.net). Start 'er up.

Unzip the irrsi archive and launch irrsi. Add the server (in this case localhost), connect to it and join the #concrete channel. You'll see that there is a concretebot in the room; everytime that runner is invoked and reaches either success or failure, the concretebot will say the room that particular run status. This can easily be expanded to spit out a ton of other information such as which build and committer started the runner. Handy enough if you don't have time to hit refresh on a web browser.

[1] https://github.com/ryankee/concrete
[2] http://nodejs.org/download/
[3] http://nodejs.org/dist/npm/
[4] http://sourceforge.net/projects/unxutils/
[5] https://github.com/bmatzelle/gow/wiki
[6] http://www.mongodb.org/downloads
[7] http://ircd.bircd.org/
[8] http://irssi.org/
[9] http://davidwalsh.name/nodejs-irc