Skip to content
Back to Blog

Continuous deployment of immutable build servers

Florian Motlik

Feb 13, 2017 10:00:00 AM

Quick iteration is key to develop a new product and finding the product-market fit. When we started Codeship it was clear that we needed to automate as much as we can. Otherwise we wouldn’t be able to succeed with a small team.

Immutable Infrastructure was important for the deployment of our test server infrastructure from the beginning. To be able to terminate and recreate our servers anytime makes our development a lot faster. It also leads to a more stable system for our customers. Listen to this talk with Chad Fowler if you want to learn more about Immutable Infrastructure.

Put together this means we can iterate quickly while keeping the server management overhead small.


How we continuously deploy and replace our build servers

Our build servers run on EC2. Whenever we start a new server it automatically connects to our build queue and starts testing our customers’ projects. Decoupling the system through a queue, in our case Sidekiq, was an easy way to make our system more resilient to interruptions.

Feature Branches and Pull Requests

Continuous Deployment was part of our development workflow from day one. All parts of our infrastructure need to be deployed automatically.

Our infrastructure currently consists of two parts. The first part, written in Ruby, is responsible for executing your build and deployment steps and stream the corresponding output back into the database. We test this part of our infrastructure with rspec and some shell scripts.

The second part takes care of building the virtual machines that are used by the Ruby scripts to run your build. We use Vagrant to provide a production-like system on every development machine.

Any changes are implemented on a feature branch and merged into master with a Pull Request. The same workflow we presented in one of our last blog posts.

Building Amazon Machine Images

Whenever a feature branch is merged we run all rspec tests on the Codeship. If they pass we start a new EC2 server and push our setup scripts into it. Through nohup we start a background process on the new server that installs and sets up everything that is necessary to run our build. It takes about an hour right now, but we are working on a quicker way to build our machines.

  # Creating the EC2 Instance
  instance = ec2.instances.create(
  :image_id => "ami-7739b41e",
  :instance_type => 'cc2.8xlarge',
  :key_name => "CloudshipCreateServerKey",
  :instance_initiated_shutdown_behavior => 'terminate',
  :block_device_mappings => {
  "/dev/sda1" => {
  :volume_size => 50,
  :delete_on_termination => true
  # Pushing scripts into the server and start building the machine
  Net::SSH.start(instance.ip_address) do |ssh|
  `tar -czf scripts.tar.gz *`
  ssh.sftp.upload!("./scripts.tar.gz", "/root/scripts.tar.gz")
  ssh.exec! "sudo mkdir /root/scripts"
  ssh.exec! "sudo tar xf /root/scripts.tar.gz -C /root/scripts"
  ssh.exec "sudo nohup time /bin/bash /root/scripts/server_setup/host/ #{ENV['AWS_KEY']} #{ENV['AWS_SECRET_KEY']} #{} &> /dev/null &"
view raw start_server.rb hosted with ❤ by GitHub

One of the first improvements will be to decouple deployment of our ruby scripts that manage the build and the virtual machine the build runs in. Tools like Docker make this possible, but there are still some issues we need to resolve. Another tool that is great for building virtual machine images and that we will transition to in the future is Packer.

After the setup is complete, we check if everything was set up correctly. To make sure all languages and tools we support are in place we have a Github repository called vm-tester. This repository connects to our supported databases and makes sure all work fine. We run this Github repository on the new server as an integration test.

When the integration tests pass we create a new AMI from the server. We use this AMI to start additional servers when our resources are saturated.

  # Create an Amazon EC2 AMI
  ec2 =
  :access_key_id => ENV['AWS_KEY'],
  :secret_access_key => ENV['AWS_SECRET_KEY'])
  ec2.images.create(:instance_id => instance_id)
view raw create_ami.rb hosted with ❤ by GitHub
Creating the AMI will restart the server. As soon as it is back online it will connect to the queue and start processing builds automatically.


Infrastructure Metrics and Server Management

We started using Librato Metrics a while ago to measure server and application metrics. We measure a lot of different metrics, from the number of currently running builds to successful/failed builds and general timing of different parts of our application.

Librato allows us to mix different app as well as server metrics to get a great overview on our current system status.

A look at our Infrastructure Metrics with Librato Metrics
Having all those metrics and the ability to add them to any part of your application is great. Whenever we see a problem with our infrastructure we make sure that there is some way we can measure it and set alarms for unexpected behavior.

Managing our server infrastructure is easy through the tools we’ve built into our admin application. We can disable or terminate any servers we have or start a new one from any AMI we’ve created. This allows us to react very quickly to any changes or problems we might see in our infrastructure.

Managing servers
We can start a new server at any time by clicking the link in our admin application:

Amazon Machine Images
Without immutable infrastructure and cloud services this workflow wouldn’t be possible.


Rebuilding our test servers regularly makes sure we start from a new and clean machine. We never have to think about configuration changes done by someone else or possible problems due to long running servers.

Going back to any old version is really only the click of a button. We can innovate quickly while knowing that if something goes wrong the impact to our customers is minimal and can be resolved quickly.

We see our decoupled and immutable infrastructure as one of our biggest assets.

Let us know how you are dealing with changes to your infrastructure and keep your server up to date in the comments.

About the Author

Florian Motlik: At Codeship I am responsible for the general tech vision and making sure that all of our users are happy and keep their build green. I've always been interested in helping people build great software, great products and just in general make something happen.

About the Codeship

The Codeshipprovides Continuous Integration and Deployment with a simple hosted system. We test every change you do in your application and if everything works we deploy to your staging or production environment. We strongly integrate with all the major Platform as a Service providers like Heroku and Engine Yard. We support GitHub, BitBucket, Amazon Web Services, Digital Ocean and many more.
Go check us out!

Written By:

Florian Motlik

Want to get our totally not sh*tty weekly newsletter?

Sometimes The Wayfarer is funny, sometimes insightful, but always the least spammy newsletter this side of Tatooine.