A few weeks ago, on a
very busy day, our continuous integration server Jenkins
was overloaded with many builds, and the build queue was too long. We decided that we need more computing power. Fortunately,
Jenkins provides a built-in ability to distribute work by adding
slave machines, so the only question was: what’s the fastest, most
cost-effective way to set up more slaves?
We
realized that some of our employee’s
personal workstations often stand idle for hours, days or even
months - either because of maternity
leaves, long vacations or part-time jobs. So we were looking for the
tools to utilize this wasted CPU power and create local Jenkins
slaves on Kenshoo employees’ personal computers.
Over a year ago we
started to use Puppet
as our automation tool to configure machines, and created a puppet
module that sets up a Jenkins slave. Since Vagrant plays nice with
Puppet (i.e. you can configure Vagrant to run a specific Puppet
module on a VM), our work was already half done - we simply used the
same puppet modules to configure the Vagrant box.
Finally we used the
Jenkins swarm
plugin that enables
automatic discovery of Jenkins slaves.
These
4 tools helped us package a Vagrant box and run “vagrant up” to
turn a developer computer into a Jenkins
slave.
The
rest of this post will describe how each of these tools were used.
We
created 2 vagrant projects.
Here
is the project vagrantFile:
The
Jenkins.pp is a starting point for the puppet process and contains
the basic Jenkins slave module and two file modules which copy
resources to the created box from the vagrant share folder:
1.
Jenkins Swarm plugin jar taken from here
2.
A service file to run the swarm jar with all necessary parameters to
connect the Jenkins master.
jenkins.pp:
swarm service:
The
Puppetfile in this project contains all the puppet modules needed for
the jenkins-slave.
We
used “librarian-puppet
install”
before running Vagrant to fetch all the puppet modules dependencies
for the jenkins-slave module into the project module folder.
Now
running “vagrant up” will create a virtual machine which will
connect to our Jenkins master and become a Jenkins slave.
So
why do we need the second Vagrant project?
We
wanted to simplify it even further for developers to run a slave.
We
used “vagrant package” to
create a box from the first project and host the .box file in one of
our local file servers for everyone to use.
All
a developer needs to do is to install Vagrant and Vritualbox,
download a simple Vagrant file and run the command line “vagrant up”.
No
need to run librarian-puppet
and puppet which can take a lot of time.
This post was authored by Alon Levi, build architect at Kenshoo.