Setting up a Salt cluster on DigitalOcean using Terraform

I use Ansible to provision or automate infrastructure tasks whenever I can. I wanted to try out Salt, from SaltStack to see how it compared. I read about its ability to push commands out to multiple servers very quickly and I wanted to try it for myself.

To gain some practical experience using Salt I used Packer and Terraform to create a Salt Master and several Salt Minions on DigitalOcean.com. This post is a guide on how to create a small Salt cluster of your own to begin learning Salt. The end result will be 1 Salt Master and 3 Salt Minions, running on the smallest DigitalOcean droplets, so you can safely experiment without a large cost.

Packer and Terraform are both great tools from Hashicorp. Packer allows you to describe a server in code, customize it and build it as an Image that you can store on cloud providers such as DigitalOcean or AWS for later use.

There are 2 overall steps to get the Salt cluster up and running. First I use Packer to create 2 base Images, a Salt master and a Minion and then I use Terraform to create the Droplets on DigitalOcean based on those images.

Packer is doing the work of provisioning the server Images ahead of time, so once the servers are running, you won’t need to SSH in to them to configure them unless there are problems.

To follow this guide, you will need:
* Install Terraform https://www.terraform.io/intro/getting-started/install.html
* Install Packer https://www.packer.io/intro/getting-started/install.html
* A DigitalOcean account and an API key https://www.digitalocean.com/?refcode=2c62404bb57

Once you have installed Terraform and Packer, the first step is to create the Images for the Salt Master and a Salt Minion using Packer.

All of the code used in this post is available on Github: https://github.com/gordonmurray/terraform_salt_digitalocean

Create a file called packer/variables.json with the following code, replace the ‘xxx’ string with your DigitalOcean API key. This will allow Packer to store the Images in your DigitalOcean account

{
"do_token": "xxx"
}

Next create a file called packer/salt_master_do_image.json with the following content:

{
  "variables": {
    "do_token": ""
  },
  "builders": [
    {
      "droplet_name": "salt",
      "snapshot_name": "salt",
      "type": "digitalocean",
      "ssh_username": "root",
      "api_token": "{{ user `do_token` }}",
      "image": "ubuntu-18-04-x64",
      "region": "lon1",
      "size": "512mb"
    }
  ],
  "provisioners": [
    {
      "type": "shell",
      "scripts": [
        "scripts/install_salt_master.sh"
      ]
    }
  ]
}

The ‘builders’ section allows you to make choices such as Image name, location, server size and type. In this example I am using Ubuntu version 18.04 and a small 512mb size Droplet.

The ‘provisioners’ section allows you to call additional scripts to install and run any commands to customize the Image further. In this example I am using a simple Bash script to install the Salt master and Salt minion. You could also use Ansible here if you wanted to.

For the Bash script to install necessary items, create a new file called packer/scripts/install_salt_master.sh with the following lines:

#!/usr/bin/env bash
set -xe
sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get install salt-master -y
sudo service salt-master status

These steps will perform an apt update on the server and install salt-master.

Next create a file called packer/salt_minion_do_image.json with the following content:

{
  "variables": {
    "do_token": ""
  },
  "builders": [
    {
      "droplet_name": "salt-minion",
      "snapshot_name": "salt-minion",
      "type": "digitalocean",
      "ssh_username": "root",
      "api_token": "{{ user `do_token` }}",
      "image": "ubuntu-18-04-x64",
      "region": "lon1",
      "size": "512mb"
    }
  ],
  "provisioners": [
    {
      "type": "shell",
      "scripts": [
        "scripts/install_salt_minion.sh"
      ]
    }
  ]
}

Create a new file called packer/scripts/install_salt_minion.sh with the following lines:

#!/usr/bin/env bash
set -xe
sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get install salt-minion -y
sudo service salt-minion status

You now have 2 Packer files which you can use to build the Salt Master and Minion Images and 2 Bash scripts to install the Salt Master and Salt Minion files.

You can validate the Packer files using :

packer validate -var-file=variables.json salt_master_do_image.json
packer validate -var-file=variables.json salt_minion_do_image.json

You can build the Images using :

packer build -var-file=variables.json salt_master_do_image.json
packer build -var-file=variables.json salt_minion_do_image.json

Once both builds have finished, in the Images section of your DigitalOcean account, you should see two new Images, ready to be used.

digitalocean imagesThose Images won’t cost you anything to store. You’ll be able to use them any time in the future as templates for any new Droplets you create. Its also possible to change their owner, so you can send Images to other DigitalOcean accounts.

Now that the Images are ready, we’ll use Terraform to create 4 new Droplets, 1 Salt Master and 3 Salt Minions.

Just like Packer, we need to give Terraform access to your DigitalOcean account, create a new file called variables.tfvars with the following content, changing the ‘xxx’ string to your DigitalOcean API key.

do_token = "xxx"

Next create main.tf:

variable "do_token" {}

# Configure the DigitalOcean Provider
provider "digitalocean" {
    token = "${var.do_token}"
}

This will tell Terraform to get the DigitalOcean Provider and use your API key.

Next create ssh_key.tf, this will tell Terraform to create an SSH key in your Digital Ocean account which can be used to SSH in to the Droplets later if needed.

module "my_ssh_key" {
  source     = "./modules/digitalocean/ssh_key"
  name       = "my_ssh_key"
  public_key = "${file("/home/ubuntu/.ssh/id_rsa.pub")}"
}

Update the path in ‘public_key’ to point to your own public key file.

Next create droplet.tf, this is where Terraform will use the Images we have created and create new Droplets.

# get the pre made salt master image
data "digitalocean_image" "salt" {
  name = "salt"
}

# get the pre made salt minion image
data "digitalocean_image" "salt-minion" {
  name = "salt-minion"
}

# generate a short random string
resource "random_string" "first" {
  length  = 8
  special = false
  upper   = false
}

# generate a short random string
resource "random_string" "second" {
  length  = 8
  special = false
  upper   = false
}

# generate a short random string
resource "random_string" "third" {
  length  = 8
  special = false
  upper   = false
}

# create salt master
module "salt" {
  source             = "./modules/digitalocean/droplet"
  image              = "${data.digitalocean_image.salt.image}"
  name               = "salt"
  region             = "lon1"
  size               = "512mb"
  backups            = "false"
  monitoring         = "true"
  ssh_keys           = ["${module.my_ssh_key.ssh_fingerprint}"]
  private_networking = "true"

  content     = "#interface: 0.0.0.0"
  destination = "/etc/salt/master"

  remote_exec_command = "salt-key -A -y"
}

# create salt minion with a unique hostname
module "salt-minion-1" {
  source             = "./modules/digitalocean/droplet"
  image              = "${data.digitalocean_image.salt-minion.image}"
  name               = "salt-minion-${random_string.first.result}"
  region             = "lon1"
  size               = "512mb"
  backups            = "false"
  monitoring         = "true"
  ssh_keys           = ["${module.my_ssh_key.ssh_fingerprint}"]
  private_networking = "true"

  # add the master IP address and the minions host name to a salt config file
  content     = "master: ${module.salt.droplet_ipv4_private}\nid: salt-minion-${random_string.first.result}"
  destination = "/etc/salt/minion"

  # restart salt-minion on the host to take in the config file change
  remote_exec_command = "sudo service salt-minion restart"
}

# create salt minion with a unique hostname
module "salt-minion-2" {
  source             = "./modules/digitalocean/droplet"
  image              = "${data.digitalocean_image.salt-minion.image}"
  name               = "salt-minion-${random_string.second.result}"
  region             = "lon1"
  size               = "512mb"
  backups            = "false"
  monitoring         = "true"
  ssh_keys           = ["${module.my_ssh_key.ssh_fingerprint}"]
  private_networking = "true"

  # add the master IP address and the minions host name to a salt config file
  content     = "master: ${module.salt.droplet_ipv4_private}\nid: salt-minion-${random_string.second.result}"
  destination = "/etc/salt/minion"

  # restart salt-minion on the host to take in the config file change
  remote_exec_command = "sudo service salt-minion restart"
}

# create salt minion with a unique hostname
module "salt-minion-3" {
  source             = "./modules/digitalocean/droplet"
  image              = "${data.digitalocean_image.salt-minion.image}"
  name               = "salt-minion-${random_string.third.result}"
  region             = "lon1"
  size               = "512mb"
  backups            = "false"
  monitoring         = "true"
  ssh_keys           = ["${module.my_ssh_key.ssh_fingerprint}"]
  private_networking = "true"

  # add the master IP address and the minions host name to a salt config file
  content     = "master: ${module.salt.droplet_ipv4_private}\nid: salt-minion-${random_string.third.result}"
  destination = "/etc/salt/minion"

  # restart salt-minion on the host to take in the config file change
  remote_exec_command = "sudo service salt-minion restart"
}

This file may look like there is a lot going on when you first look at it, though the the sections are straight forward. The ‘data’ section at the beginning of the file is getting the pre-made Images we made earlier in to variable names we can reference later in the file.

The ‘resource’ sections are creating random strings, so that we can give the Salt Minions unique host names, otherwise they’ll have the same host name when we create them.

The ‘modules’ are where the real work is happening. Each module section is a Droplet that Terraform will create. There is 1 for a Salt master and 3 for Salt Minions. The module is describing the Droplet that will be made, including information such as the Image, the host name, region, public key for access and the size of the Droplet to use.

Two lines worth highlighting though are :

 # add the master IP address and the minions host name to a salt config file
 content = "master: ${module.salt.droplet_ipv4_private}\nid: salt-minion-${random_string.first.result}"
 destination = "/etc/salt/minion"

Those lines are applied to the Minions only and they are used to create a file called /etc/salt/minion within each Minion. This file will contains both the Private IP address of the Salt master Droplet so they know what to connect to, and also the unique hostname of the Minion itself, otherwise each Minion will contact the Master under the host name of ‘salt-minion’

All 4 Modules also contain a ‘source’ field and each one points to a single module in the modules folder called modules/digitalocean/droplet/main.tf. Creating Terraform Modules allows us to create re-usable blocks of code we can use in other Terraform projects.

resource "digitalocean_droplet" "default" {
  image              = "${var.image}"
  name               = "${var.name}"
  region             = "${var.region}"
  size               = "${var.size}"
  backups            = "${var.backups}"
  monitoring         = "${var.monitoring}"
  ssh_keys           = ["${var.ssh_keys}"]
  private_networking = "${var.private_networking}"

  provisioner "file" {
    content     = "${var.content}"
    destination = "${var.destination}"
  }

  provisioner "remote-exec" {
    inline = [
      "${var.remote_exec_command}",
    ]
  }
}

# Send outputs from this resource back out
output "droplet_ipv4" {
  value = "${digitalocean_droplet.default.ipv4_address}"
}

# Send outputs from this resource back out
output "droplet_ipv4_private" {
  value = "${digitalocean_droplet.default.ipv4_address_private}"
}

We need to create 1 more filemodules/digitalocean/ssh_key/main.tf, a Module for the Key pair we used earlier:

resource "digitalocean_ssh_key" "default" {
  name       = "${var.name}"
  public_key = "${var.public_key}"
}

# Send outputs from this resource back out
output "ssh_fingerprint" {
  value = "${digitalocean_ssh_key.default.fingerprint}"
}

Our next step is to initialise Terraform and run its ‘plan’ command to see what Terraform will create:

terraform init
terraform plan

This command will show you and output consisting of 5 items to create, the public key, the Salt Master and 3 salt minions.

To go ahead and create them, run:

terraform apply

This will show you the same output as the ‘plan’ command again with a N/Y Prompt for it to go ahead and create the Droplets, you can press Y to continue or you can use the following to skip the confirmation and continue to build the Droplets

terraform apply -auto-approve

In a short few minutes, you will now have a Salt Master and 3 Salt Minions running ready to you.

To determine the IP addresses of the new Droplets, you can look at your DigitalOcean Droplets page, or you can type:

terraform show

This will show you the IP adresses and other information about the Droplets Terraform has created.

To log in to the Salt Master, you can use:

ssh root@{ip address of salt master}

As you are using a Public key, you will be able to log directly in.

You can issue the following command to see the Salt Minions that are tring to connect to the Master:

salt-key -L

You can accept all Minions using:

salt-key -A -y

Your Salt Master can no issue commands to each of the Minions, for example:

salt * cmd.run 'hostname -a'

You will get back a response from each Salt Minion with their unique host name.

You now have a small, functioning and affordable Salt cluster to continue to use to learn about Salt, grains, pillars and Services.

All of the code used in this post is available on Github: https://github.com/gordonmurray/terraform_salt_digitalocean

Leave a Reply

Your email address will not be published. Required fields are marked *