Graeme Matthieson: Automate All The Things

Published: April 13th 2013by Orde Saunders

Graeme Matthieson (@mathie) was talking about building your development and production environments with vagrant and puppet at Whisky Web II, these are my notes from his talk.

Why?

Apply configuration quickly
Handles dependencies
Applies configuration consistently
Mimic production in staging - especially useful for QA testing
Mimic production in development

This makes it very easy to test scenarios at any stage of developement and release cycle and reliably apply.

Why not?

Up front effort - can take time to set up (days)
You have to be much more rigorous - one mistake goes everywhere

Which one?

(In the order I tried them)

LCFG: Local configuration system. Now LCFG2 and still actively maintained.
CFEngine: 1983, one of the first open source systems and still actively developed. Theoretically very good.
Puppet: Spawned from frustration with CFEngine, written in Ruby. Thick server - thin client.
Chef: All the cool kids like this. Spawned from frustration with Puppet. Thin server - thick client.

All have their pros and cons.

Vagrant

Vagrant is kind of neat. Turned up in the last two years and started as a way of manging Virtual Box VMs. Now supports different VM providers including AWS.

Controlled through a vagrantfine which is pure Ruby.

Start with a very basic install with next to no config.
Create network configs including port forwarding.
Share folders between host and VM.
Provisioning (Puppet or Chef): use the shell to run commands using shell scripts it uploads.
Multiple views: Can run multiple VMs to model an entire production environment.
Vagrant config is Ruby: You can use Ruby to do things.
Manage the cluster: Boot, Destroy, Pause, Resume, SSH.

Bootstrapping

This is the awkward bit, you are starting with a new machine with nothing on it. Gets as far as being able to run the Puppet agent.

Set the host name
Set the Puppet master's IP address so we don't need DNS
Install Puppet - latest stable, some distros are quite out of date
Get the Puppet repository
Run Puppet apply to load a minimal configuration from a local file that lets it talk to the master
The nodes then get their configuration from the Puppet master

Nodes

The node tells the Puppet system what it's supposed to do - i.e. what type of server this is: app, database, &c.

Classes can be used to group configurations of nodes.

Modules

Groups of classes are modules that then provide a service. Can define which modules and versions to install and how to configure them. Can get complex if you try to generalise them across multiple platforms.

Resources

This is what Puppet can manage.

Files
Permissions
Content
Services
Configuration
Ensure they're running
Packages
Install or remove
Manage versions
What ever you can think of and code

There are providers for each system that will abstract the operating system away from the config. It's not perfect as package names can vary for the same thing which require conditionals in the config.

Puppet at work

Puppet master provides information to Puppet clients that run a service that checks the master periodically over SSL connections. The Puppet master compiles the list of information that a client needs so it only gets the minimum it needs. Once it has a configuration it checks to see if it has changed since last time and, if so, runs the changes.