Become a Jenkins Slave

A few weeks ago, on a very busy day, our continuous integration server Jenkins was overloaded with many builds, and the build queue was too long. We decided that we need more computing power. Fortunately, Jenkins provides a built-in ability to distribute work by adding slave machines, so the only question was: what’s the fastest, most cost-effective way to set up more slaves?

We realized that some of our employee’s personal workstations often stand idle for hours, days or even months - either because of maternity leaves, long vacations or part-time jobs. So we were looking for the tools to utilize this wasted CPU power and create local Jenkins slaves on Kenshoo employees’ personal computers.

We chose Vagrant - a tool to easily build virtual machines over Oracle’s Virtualbox.

Over a year ago we started to use Puppet as our automation tool to configure machines, and created a puppet module that sets up a Jenkins slave. Since Vagrant plays nice with Puppet (i.e. you can configure Vagrant to run a specific Puppet module on a VM), our work was already half done - we simply used the same puppet modules to configure the Vagrant box.

Finally we used the Jenkins swarm plugin that enables automatic discovery of Jenkins slaves.

These 4 tools helped us package a Vagrant box and run “vagrant up” to turn a developer computer into a Jenkins slave.

The rest of this post will describe how each of these tools were used.

We created 2 vagrant projects.

The first creates a jenkins-slave from a basic Ubuntu box taken from http://www.vagrantbox.es/

Here is the project vagrantFile:

The Jenkins.pp is a starting point for the puppet process and contains the basic Jenkins slave module and two file modules which copy resources to the created box from the vagrant share folder:

1. Jenkins Swarm plugin jar taken from here

2. A service file to run the swarm jar with all necessary parameters to connect the Jenkins master.

jenkins.pp:

swarm service:

The Puppetfile in this project contains all the puppet modules needed for the jenkins-slave.

We used “librarian-puppet install” before running Vagrant to fetch all the puppet modules dependencies for the jenkins-slave module into the project module folder.

Now running “vagrant up” will create a virtual machine which will connect to our Jenkins master and become a Jenkins slave.

So why do we need the second Vagrant project?

We wanted to simplify it even further for developers to run a slave.

We used “vagrant package” to create a box from the first project and host the .box file in one of our local file servers for everyone to use.

All a developer needs to do is to install Vagrant and Vritualbox, download a simple Vagrant file and run the command line “vagrant up”.

No need to run librarian-puppet and puppet which can take a lot of time.

This post was authored by Alon Levi, build architect at Kenshoo.