Updating Windows and Linux servers

Patch management is critical to keep your systems up to date. To ensure ongoing performance and to avoid issues such as unpatched vulnerabilities. In this post, you will gain practical experience in patching operating systems such as Windows Server and Linux based machines in both a manual and automated way using tools such as Ansible.

We will cover the following topics:

  • Patching Windows Servers
  • Patching Windows Servers using Ansible
  • Patching Linux servers
  • Patching Linux servers using Ansible
  • Replacing servers with pre-patched servers
  • Performing updates to AWS Relational Database Service (RDS) instances

Patching Windows Servers

Microsoft windows provides a service called Windows Update which is designed to keep your operating system up to date with the latest bug fixes or security patches supplied by Microsoft.

Windows Update is usually enabled by default on a Windows PC. It is important to enable this service and allow it to keep your system up to date. In our next section we will make sure Windows Update is enabled and functioning.

Enable Windows Update to install updates automatically

From your windows desktop, use the following steps to open Windows Update and ensure it is set up to install Windows Updates automatically:

  1. Press on Start | select Control Panel
  2. Once the Control Panel window has opened, click on System and Security
  3. Once the System and Security window has opened click on Windows Update
  4. Once the Windows Update window has opened, click on Change settings to the left of the screen:
  5. In the Important updates make sure that Install updates automatically (recommended) is selected.
  6. Using the Install new updates drop down menu, you can choose when to install new updates. Select a frequency and time that works for you, such as Every day at 06:00.
  7. Using the Recommended updates check box, you can choose to allow Windows to also install recommended updates along with important updates.
  8. Using the Who can install updates, you can choose to allow any users of your windows machine to install updates.
  9. Once you have made your choices, press on OK to apply.

With these changes in place, Windows will check for updates and keep itself up to date.

In our next section we will look at how to install Windows Updates manually.

Installing Windows Updates manually

There may be times when you would not like to enable automatic Windows updates and so you should apply any Windows updates manually.

  1. Press on Start | select Control Panel
  2. Once the Control Panel window has opened, click on System and Security
  3. Once the System and Security window has opened click on Windows Update
  4. Press on Check for updates on the left. Windows will spend some time searching for any updates that need to be applied as shown in the following screenshot:
  5. If you would like to go ahead, press on Install Updates. Windows update will begin downloading and installing one or more updates. The Window might close as it performs the updates in the background or it may stay open showing current progress.

Windows update installing an update In this section, we have covered a number of steps to ensure that the Windows Update service is running and is installing any necessary updates. In our next section, we will look at how to handle failed windows updates which may occur. Handling Failed Windows updates

When installing Windows updates, unfortunately some updates can fail to apply. Sometimes pressing on Try again can work in case there was a temporary connectivity issue that interrupted the download of a Windows Update.

If the problem persists, there is a Get help with this error link which can open a Help page that may help you do see the problem and fix it.

Another option when there is a failed update is to click on View update history to open a list of all updates applied as shown in the following screenshot:

In the list, look for updates with a Status of Failed. Take a note of the value starting with KB which stands for Knowledge Base and perform a search online for the value, such as KB915597. There will often be a result from a Microsoft Community forum at https://answers.microsoft.com

In this section, we looked at ensure that the Windows Update service is running well and also how to handle any failed Windows updates. This is a manual process which won’t work well if you have many servers. In our next section we will look at patching Widows server using Ansible to help automate the process.

Patching Windows Servers using Ansible

We will need to perform a number of steps first to configure our Windows server so that Ansible can connect and perform any tasks. Our first step is to make sure that remote connections are allowed to connect to the Windows server.

Allow Remote Desktop

On the Windows Server, click on Run and search for View Advanced System Settings and click the search result to open it.

Navigate to the tab called Remote and in the Remote Desktop section, click on Allow connections from computers running any version of Remote Desktop and press on OK.

Next, we will need to make sure that PowerShell, used by Ansible is up to date and that Windows is set up to allow Ansible to connect.

The following steps are taken from the Ansible documentation available at: https://docs.ansible.com/ansible/latest/user_guide/windows_setup.html. Also, Ansible requires that Powershell version 3.0 at a minimum is running on the target Windows machine.

Update Powershell

On the Windows machine, open up Windows PowerShell ISE as Administrator. Paste in the following content. Include the username and password that you usually use to log in to Windows and press on Run:

$url = “https://raw.githubusercontent.com/jborean93/ansible-windows/master/scripts/Upgrade-PowerShell.ps1”
$file = “$env:temp\Upgrade-PowerShell.ps1”
$username = “Administrator”
$password = “Password”
(New-Object -TypeName System.Net.WebClient).DownloadFile($url, $file)
Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Force
# Version can be 3.0, 4.0 or 5.1
&$file -Version 5.1 -Username $username -Password $password –Verbose

This PowerShell script will update PowerShell and change the Execution Policy in Windows to allow Ansible to operate. Depending on your system, this process may take a few minutes to complete.

Also, a warning that your Windows server will reboot automatically once it has upgraded Powershell so it is best to perform this step at a time you are comfortable for your Windows server to reboot.

Configure Windows for Ansible

The next step is to configure the Windows server for ansible

Download the following Powershell script from Ansible’s Github account and save it as locally as ConfigureRemotingForAnsible.ps1.

This script checks the current WinRM (PS Remoting) configuration and makes the necessary changes to allow Ansible to connect, authenticate and execute PowerShell commands. https://raw.githubusercontent.com/ansible/ansible/devel/examples/scripts/ConfigureRemotingForAnsible.ps1

You will be asked if you trust this file and if you would like to prevent it running. Press r and Enter to allow the script to run once.

If your Windows Server is hosted with AWS, you will need to open Port 5986 for HTTPS-WinRM in the Security Group used by your Windows server.

To test that Ansible can connect to the Windows host, run the following from your terminal:

ansible windows -i hosts -m win_ping

You should get back a result showing a Success message as shown in the following screenshot:

Once Windows is configured, we can begin to create our Ansible playbook.

Create a new file called windows.yml with the following content for our Playbook:


– hosts: windows
gather_facts: no
roles: – windows-update

This is our main playbook file and it will call a role called windows-update.
Next, we will create our main tasks file. Create a new file called roles/windows-update/tasks/main.yml with the following content:


– name: Search and download Windows updates without installing them
win_updates:
state: searched

This task will tell Windows to search for Updates, but not install any yet.

The final step is to create a file called Hosts, this will contain a list of one or more IP addresses of the Windows servers that you would like to update:

Hosts

[local]
localhost
[windows]
192.168.1.100

You will need to change the IP address in the hosts file to match the IP address of your Windows server.

If you are running Ansible from your own Windows desktop, you can omit the windows section and update your playbook to use hosts: local instead of hosts:windows to run the playbook locally on your machine.

Run the playbook using the following command:

Ansible-playbook windows.yml -i hosts

Next, let’s update the playbook to get Windows to install Security and Critical updates only. Edit the main.yml file and append the following:

– name: Install all security, critical, and rollup updates
Win_updates:
Category_names:
– SecurityUpdates
– CriticalUpdates
Reboot: yes

The Reboot: yes option will reboot the windows server if it is necessary to complete any Windows updates.

Run the playbook again using the following command:

Ansible-playbook windows.yml -i hosts

Congratulations! You have successfully written an Ansible playbook to keep one or more Windows servers up to date.

So far we have covered performing Windows updates both via manual steps and also in an automated way using Ansible. In our next section we will change to Linux servers and see how we also perform updates both manually and via Ansible.

Patching Linux servers

In this section we will look at some of the command line tools we can use to keep our Ubuntu based servers or desktops up to date. We will use Ubuntu’s package manager tool called Advanced Packaging Tool (APT) to perform the updates needed.

In your terminal use the following command. It will not make any changes to our system, it will only download package information from all configured sources:

apt update

This will download package information from sources in /etc/apt/sources.list but it will not perform any actual updates.

Next use the following command in your terminal to upgrade existing packages:

apt upgrade

Upgrade will read the updated package information and it will perform the updates to bring your packages up to date. It will provide a list of packages it will upgrade for you to approve before upgrading.

The next command in our terminal will update any dependencies:

apt dist-upgrade

A dist-upgrade is similar to an upgrade, however it can also handle any dependencies such as downloading additional items that an upgrade requires and may also remove older dependencies.

Now that we have performed our updates, we can perform some clean up steps. Use the following command in the terminal to clean up references to unnecessary packages

apt autoclean

autoclean will remove the local cache of retrieved package files that are no longer in use.

Use the following command in your terminal to fully remove any unnecessary packages:

apt autoremove

autoremove will remove any packages that were installed automatically as part of a dist-upgrade that are no longer needed.

We can put the above commands in to one script. Create a new file called update.sh with the following commands:

apt update
apt upgrade -y
apt dist-upgrade -y
apt autoclean -y
apt autoremove -y

The -y parameter will allow the command to continue without waiting for your user input.

While the above commands will ensure your linux based system is up to date with the latest packages, it is often not a good idea to run all of those commands on a live production server. You may have software on your live server that will no longer run well if packages are upgraded. A good example of this is if you have a web server using PHP and you upgrade the PHP version to the latest version, your application may stop working.

While it is possible to reverse some changes after performing upgrades, a better approach is to test that your application works on an upgraded system first. Later in this post we will look at approaches to handle this.

To run our update.sh script, run the following command:

sudo bash update.sh

This script will run all of the commands to update, upgrade and clean our locally installed packages.

When using an Ubuntu based system, there is an additional command that can be run to see if there is an update to the operating system you are running currently. If you are running a system with Ubuntu 18.04 LTS installed, the following command will show you that Ubuntu 20.04 LTS is available and will perform the upgrade for you if it is possible to do so based on the packages you have installed.

sudo do-release-upgrade

This command will upgrade Ubuntu to the latest version of the operating system that is available. A configuration file can be changed if you would like to stop release upgrades, or if you’d like to only upgrade to Long Term Support (LTS) versions of Ubuntu, or use a normal release upgrade to get the latest available version of Ubuntu.

To change the configuration file, you can change the ‘Prompt’ value in the file called /etc/update-manager/release-upgrades to ‘normal’, ‘lts’ or ‘never’ to suit your own preferences. ‘Lts’ is usually a good option to get the latest long term supported version.

Now that we know how to update a single Ubuntu server, in our next section we will use Ansible to perform the same overall steps in an automated process that could be used to update many servers automatically.

Patching Linux servers using Ansible

In this section, we will use Ansible to perform the same steps to update a linux server as we performed earlier to bring a server up to date.

We will use the folder structure of an Ansible Role, so that the finished Role can be used directly or easily added to our existing Ansible project from an earlier post in the ‘Using Ansible’ section.

First we will create the main file in our Ansible Role. It will instruct Ansible to run the role called update on the hosts in the all group and to connect using the user called ubuntu.

In your editor, create a new file called 03_update.yml with the following content:


– hosts: all
gather_facts: no
become: yes
user: ubuntu
roles:
– update

The field become: yes is applied as we will need Sudo permissions to perform the APT updates.

The gather_facts: no is applied as we do not need to gather information from the target host for this role and it allows the Playbook to run a little faster.

Next we will create the main file that will be used by our Ansible role. In your editor, create a new file called /roles/update/tasks/main.yml with the following content:

– name: Run apt update
apt:
update_cache: yes

– name: Run apt upgrade
apt:
name: “*”
state: latest

– name: Run apt dist-upgrade
apt:
upgrade: dist

– name: Run apt autoclean
apt:
autoclean: yes

– name: Run apt autoremove
apt:
autoremove: yes

This is our main tasks file where we will perform the same APT update steps as we have performed manually earlier in this post. We do not need any other files or templates for updating the target server, so the two files we have just created are enough to continue.

If you have kept your webserver creating using Ansible running from ‘Using Ansible’ then you can use the following command to run your playbook to update your webserver:

Ansible-playbook 03_update.yml –ask-vault-pass

Alternatively, instead of creating the new 03_update.yml paybook, you could edit your existing webserver.yml file and add the update role to the existing roles so that the updates would be performed before installing apache or PHP, so your webserver.yml file would become:

Roles:
– update
– apache
– php

Congratulations! You have successfully written an Ansible playbook to keep one or more Ubuntu servers up to date.

Safely updating production servers

In some cases it can be quite damaging to perform updates to an active server in production use. If a server contains a number of different packages, one or more of them might be updated and reach a state where they are no longer compatible with each other, potentially causing your software to stop working.

One approach to safely update a production server is to use one or more tools to create a new server, provision it with the exact same software and perform any updates there first. Once confirmed, the new server can replace the old server by reassigning its IP address or DNS values so it becomes the new production server. An earlier post includes a step by step guide on how to create a new server in this way in the section called Replacing a webserver using Packer, Ansible and Terraform.

Another approach that can be taken to avoid updating a server directly if you have not yet taken steps into automating your builds and deployments is to make a copy of an existing server. Once copied, perform the updates on the copy of the server first to make sure it is safe. Once you are sure the update process is safe, you could then apply the same update steps to the production server.

Replacing servers with patched servers earlier in this post we learned how to apply updates to an existing Windows or Linux server. We also learned that applying updates to a production server – that is, a server that is in active use may cause problems for users.

In this section we will use the AWS console and the AWS CLI to copy or clone an existing server to create an Amazon Machine Image (AMI).

This AMI can serve simply as a backup. This AMI can also be used as a template for testing any updates in isolation.

The AMI can also completely replace an existing server once it has been updated successfully.

Clone an EC2 instance using the AWS console

In the following steps we will use the AWS console to clone an existing EC2 instance to make an AMI available for testing or replacing.

  1. To begin creating an AMI from an existing EC2 instance, log in to your AWS Console and click on EC2 in the Compute section or by searching for EC2 in the Find Services search bar.
  2. Click on Running Instances to view any existing EC2 instances. If you were following along with earlier posts and left your webserver running, you should be able to see at least one EC2 instance.
  3. Next to an existing EC2 instance, select the checkbox and then click on the Actions drop down menu and select on Image and then Create Image.
  4. In the window that opens, you can give your image a name such as webserver, a description and a checkbox to prevent the EC2 instance from rebooting. Fill in each field to give your AMI a suitable name and a relevant description and select No Reboot
  5. In the same window, a shot list of disk volumes that are attached to your EC2 instance will be visible. You can leave this list unchanged unless you want to make specific changes to the volumes such as changing the disk size ( Note that if you change the disk size here, it will only change the disk size of the new EC2 instance that you create, it will not change your existing EC2 instance).
  6. When you are ready, click on Create Image. This will start the process of creating a copy of your existing EC2 instance.
  7. You can click on the View pending image link to see your AMI being created, or if you have already closed the window you can click on AMIs under Images from the navigation mentioned on the left of the screen.

Your AMI will remain in a Pending state for a short time though this will vary depending on the size of your EC2 instance.

We have now created an AMI using the AWS Console using a manual process. In our next section, we will see how to perform the same steps using the AWS CLI as the CLI approach will allow us to automate this process if needed.

Clone an EC2 instance using the AWS CLI

The steps we have just taken have created an AMI of our existing EC2 instance. This AMI can be used in a number of ways. If our existing EC2 instance were to become damaged in some way, this AMI could be used to create a new copy of the EC2 instance. The AMI can be used to copy

our Webserver and make it available to another geographical region in your AWS account or even make your AMI available to another AWS account.

To perform the same action of creating an AMI using the AWS CLI, you could use the following cmmand:

aws ec2 create-image –instance-id {you-instance-id} –name webserver-cli –no-reboot –region us-east-1

If you are unsure of the instance ID of your existing EC2 instance, you can view it in the AWS Console or use the AWS CLI to describe the EC2 instance as follows:

aws ec2 describe-instances –region eu-west-1 –filter ‘Name=tag:Name,Values=webserver’

This will give you a detailed output of the EC2 instance. You can limit the information returned to just show the tag name and instanceId as follows:

aws ec2 describe-instances –query ‘Reservations[*].Instances[*].[Tags[0].Value , InstanceId]’ –region us-east-1 –filter ‘Name=tag:Name,Values=webserver’

Once you create an image, the output will show an AMI ID.

Now that we have created an AMI, in our next section we will learn how to turn this image in to a working EC2 instance.Create an EC2 instance from an AMI using the AWS console

Earlier in this post we created an AMI from an existing EC2 instance. In this section we will reverse the process to create a new working EC2 instance from our AMI. Once the new EC2 instance is running, we can perform the updates we performed earlier in a safe environment that will not impact on users.Use the following steps to create an EC2 instance from the AMI:

  1. To begin creating a new EC2 instance from an AMI, log in to your AWS Console and click on EC2 in the Compute section or by searching for EC2 in the Find Services search bar.
  2. From the navigation menu on the left, select Images | AMIs
  3. This section will list any AMIs already in your account. If you followed the steps from the previous section, you should have at least one AMI called webserver in the list.
  4. Select the Webserver AMI using the checkbox to the left of the AMI and click on Launch.
  5. You will be asked to select an Instance Type. Select t2.micro for a small low cost server if you are unsure of the type of server to use and click on Next: Configure Instance Details (It is tempting to press on Review and Launch, however this will make a number of assumptions and you may end up with a Webserver that you cannot connect to).
  6. In the Configure Instance section, select the VPC you would like to use from the Network drop down menu and press on Next: Add Storage.
  7. In the Add Storage section, you can leave the existing choices in place, unless you would like to give the new server some larger disk space and click on Next: Add tags
  8. In the Add Tags section add any option tags you would like to use here. For example, a Tag with a key of Name and a Value of Webserver – New. Once you are ready to continue press on Next: Configure Security Group.
  9. In the Security Group section, select on Select an existing security group and in the list look for the webserver-security-group we created earlier in the book when creating the webserver. Once you are ready to continue press on Review and Launch
  10. On this Review page you can view your choices and when you are ready to continue press on Launch
  11. On the modal window that pops up, select Choose and existing pair from the first drop down menu. From the second drop down menu, select ansible-webserver from the list which we created earlier. You can also create a new Key pair by choosing Create a new key pair if you wish. This will allow you to create a new key pair and download it. Make sure you have selected I acknowledge that I have access to the selected private key file and press on Launch Instances.
  12. The next screen will let you know that your instance is being created. Press on View Instances to go back to the main list of your EC2 instances and you should now see a new EC2 instance called Webserver – New if you filled in the optional Tags in the earlier step.
  13. Take a note of the IPv4 Public IP address of the new webserver so that you can connect to the instance

Once the Instance State shows Running, you can now connect to the instance using Remote Desktop Protocol (RDP) if it is a Windows based server or using SSH if it is a Linux based server.

Once you are connected, you can perform one of the actions above to perform the updates on your server using the manual or automated approach using Ansible.

When you have your new instance updated successfully, you will know if the updates are safe to apply to your original server in production. You could also start to use this new server instead

and update your domain name or DNS records to use this server instead and remove the old one.

In the same way se used the AWS Console and then the AWS CLI to create an AMI. In our next section, we will use the AWS CLI to create an EC2 instance using the AMI.

Create an EC2 instance from an AMI using the AWS CLI

Creating an EC2 instance from an AMI is a little different, you’ll need to know a couple of details first before you can create the instance.

The following API commands can be run in a single line, the line breaks represented by the backslash \ is to make the examples easier to read and will also work in the command line.

You will need to know the AMI ID and your existing security group ID before we can continue.

To use the AWS API to determine the ID of your AMI, use the following command:

aws ec2 describe-images \
–owners self \
–filters “Name=name,Values=webserver”

This command will return a JSON object with the full details of the Image we created earlier. You will be able to see the AMI ID within the JSON but if you’d like to get the AMI ID only, you can use the query parameter to reduce the information returned as follows:

aws ec2 describe-images \
–owners self \
–filters “Name=name,Values=webserver”
–query ‘Images[*].{Image:ImageId}’

In the output, you will see a short string in the format of ami-abc1234567890. The actual value you see will be slightly different but copy it for now and we will use it shortly.

To use the AWS API to determine the ID of your existing security group, use the following command:

aws ec2 describe-security-groups \
–group-names webserver-security-group \
–query ‘SecurityGroups[*].{Group:GroupId}’

Take note of the value that is returned, it will look similar to sg-1234567890.

Once you have your AMI and Security Group ID, add those details to the following command and run the completed command to start the new EC2 instance:

aws ec2 run-instances \
–image-id [add your AMI ID here] \
–instance-type t2.micro \
–key-name ansible-webserver \
–security-group-ids [add your security group ID here]\
–tag-specifications ‘ResourceType=instance,Tags=[{Key=Name,Value=Webserver-CLI}]’

In the output, take a note of the InstanceId value. We will use this value to get the IP address of the new instance so we can connect to it, it will look something like i-1234567890

Use the following command to retrieve the public IP address of the new instance:

aws ec2 describe-instances \
–instance-id {instance-id} \
–query ‘Reservations[*].Instances[*].NetworkInterfaces[*].Association.PublicIp’

You can now connect to the instance using RDP if it is a Windows based server or using SSH if it is a linux based server.

Once you are connected, you can perform one of the actions above to perform the updates on your server using the manual or automated approach using Ansible.

If you would like to remove your new EC2 instance after you have finishing using it, you can run the following command:

aws ec2 terminate-instances –instance-ids {instance-id}

In this section we used the AWS CLI to take a copy of an existing server in the form of an AMI. We used this AMI to create a new working server to use to test our updates. This can be a lengthy process, but it can help provide a safe place to test your updates before applying to production servers.

In our next section we will move away from working on EC2 instances and instead take a look at performing updates to AWS Relational Database Service (RDS) instances.

Performing updates to AWS Relational Database Service (RDS) instances

AWS RDS provides two different mechanisms to help you to keep your RDS instances up to date in relation to security or version updates.

The first is called Auto Minor version and is an optional parameter you can apply to your RDS instances. If you are using a particular version of mySQL such as version 5.7.21 and a new minor version such as 5.7.22 becomes available, then the RDS instance will automatically update to the next version during your defined Maintenance window.

The next mechanism is the Maintenance Window. This process allows you to choose a day of the week and a period of time within that day to allow Maintenance to occur automatically. This kind of maintenance can be the auto minor version we covered a moment ago, but it can also be a larger upgrade you have chosen to perform such as a mySQL version change from version 5.7.22 up to version 8.0 or increasing disk space, or changing the class of the server from an M5 to and R5. It is even possible to make several changes in one maintenance window.

The maintenance window allows you to control when these changes occur. It is recommended to perform changes such as this at an off-peak time when your database is not in use or is receiving minimal usage.

RDS also contains an ability called Multi Availability Zones (AZ) which is a process whereby RDS will create a copy of your RDS instance in a separate zone for high availability. This means if your primary RDS instance experiences some failure, the other instance can take over.

MultiAZ is not directly related to performing updates, however it can facilitate applying updates minimizing interruption for your database users. If you have MultiAZ enabled and you have chosen to perform a significant update to your RDS instance during a maintenance window, then RDS will apply the changes to the standby instance and once it is ready, RDS will ‘fail over’ to this updated instance, minimizing the time that your database users may be interrupted.

Summary

In this post, we learned a number of different approaches on how to perform updates to Windows or Linux based servers. We looked at how to perform those updates manually as well as perform the same steps in code using Ansible. We also looked at how to clone existing instances to use them as a platform for testing updates first or to become replacement instances instead. We also looked at the options available to us when performing updates to AWS Relational Database Service (RDS) instances.

In our next post, we will review a number of ways to monitor our infrastructure once it is running

Learn how to back up and validate your database backups

In this post we”ll focus on backing up our application database. Most web sites or web applications will be backed by some form of database. It is important to know how to create, query and backup your database in case anything happens to your data and you need to restore it.

It’s also important to validate your database backups. If you experience significant database issues, you don’t want to find out at the last minute that your nightly backups have not been working correctly. In this post, we will first create database backups with Mysqldump and then test our database backups. We will then learn to use RDS Snapshots and also use AWS S3 for storing backups.

We will cover the following topics:

  • Database backups with Mysqldump
  • Testing our database backups
  • Using RDS Snapshots
  • Using AWS S3 for storing backups

Database backups with Mysqldump

While connecting to a database, you’ll need 4 details:

  • A host address
  • Database name
  • Username
  • Password

The host address will vary a lot depending on where your database is located. If you are backing up a database on your local machine, it could be ‘localhost’ or if you are backing up a database in an RDS instance, the host address could look something like this:

mydatabase.abc123.us-east-1.rds.amazonaws.com

If you would like to install a database locally to use some of the commands in this post, you can install MariaDB Server which is a popular relational database used in web applications by using the following command in your terminal:

sudo apt install mariadb-server

This will install mariadb server locally on your machine. By default, you will be able to log in to the database using the following command in your terminal:

Sudo mysql –u root

Once you have logged into the database, you can create a new database user that you can use in the examples in this post using the following commands:

CREATE USER ‘user’@’%’ IDENTIFIED BY ‘mypassword’;
GRANT ALL PRIVILEGES ON *.* TO ‘user’@’%’;
FLUSH PRIVILEGES;

The commands above will create a user called user with a password of mypassword. It will then provide the user with permissions to query the database and finally it will flush the privileges to make sure the user and permission changes have been applied.

For local testing and development, the above GRANT command provides all privileges to the new database user. In a production environment it is recommended to give reduced permissions to a database user that is used in a web application to help to avoid problems such as deleting data or dropping tables.

A more suitable GRANT command might look like:

GRANT SELECT, INSERT, UPDATE, ALTER, CREATE TABLE, INDEX ON *.* TO ‘user’@’%’;

Before we can begin to backup our data, we need to install a program called mysqldump first before we can use it.

On Ubuntu, you can install it using the command:

sudo apt install mysql-client

Once it is installed, we can start backing up a database using the following (Substituting in your database host address, database name, username and password):

mysqldump –host=${DATABASE_HOST} –user=${DATABASE_USERNAME} –password=”${DATABASE_PASSWORD}” ${DATABASE} –single-transaction > ${DATABASE}.sql

This command may take some time, depending on the size of your database and your connection to your database and it will create a file with the same name as your database with a .sql extension.

If you need to create a sample database with some sample data for the examples in this post, you can use the following:

Create a new file called create.sh with the following content:

#!/usr/bin/env bash
# The following assumes you have mysql or mariadb installed locally
# It will create a database and import the data from the seed file.

DATABASE_HOST=”localhost”
DATABASE_USERNAME=”user”
DATABASE_PASSWORD=”password”
DATABASE=”myapplication”

# create a local database called ‘validate’
mysql –host=${DATABASE_HOST} –user=${DATABASE_USERNAME} –password=”${DATABASE_PASSWORD}” -e “create database if not exists validate;”

# import the sample users table and data
mysql –host=${DATABASE_HOST} –user=${DATABASE_USERNAME} –password=”${DATABASE_PASSWORD}” ${DATABASE} < seed_data.sql

You will need to update the database variables at the beginning to suit your database host, username and password.

Next we will create a very small sample set of data. Create a new file called seed_data.sql with the following content

CREATE TABLE users ( id int unsigned not null auto_increment, name varchar(20) not null );
INSERT INTO users ( id, name ) VALUES ( null, ‘Jane Doe’ );
INSERT INTO users ( id, name ) VALUES ( null, ‘John Doe’ );

The above files will create a database called validate and will import a sample users table.

Backing up data on remote hosts

If you are backing up a database from RDS, the backup process will be much quicker when running from an EC2 instance in the same region. You can use a command line took called SCP (short for OpenSSH secure file copy) or similar once the backup has completed to copy the backup from the EC2 instance down on to your local machine.

Sometimes you might want to only backup a specific table within a database, or to skip a single table if it is too large. You can pass additional parameters to mysqldump when you are running it in your terminal to help with those situations.

To back up specific tables, in your terminal use the following command:

mysqldump –host=${DATABASE_HOST} –user=${DATABASE_USERNAME} –password=”${DATABASE_PASSWORD}” ${DATABASE} table1 table2 table3 –single-transaction > ${DATABASE}.sql

You will need to update the mentions of table1, table2, table3 to the names of tables in your database.

To skip a specific table, in your terminal use the following command:

mysqldump –host=${DATABASE_HOST} –user=${DATABASE_USERNAME} –password=”${DATABASE_PASSWORD}” ${DATABASE} –single-transaction –ignore-table=${sometable} > ${DATABASE}.sql

If your backup file is a large file, you could compress it using the following command:

gzip ${DATABASE}.sql

This should reduce the size of the file quite a bit and make it easier and faster to copy elsewhere.

Once we have a good process for backing up our database, an important next step is to test those database backups often. Your back up process would be no good to you or your business if one day you needed a backup in an emergency and found out too late that the backup process was not working or not backing up all of the data correctly.

In our next section, we will look at a way to retrieve our backup, restore it to a working database and test it to make sure it has performed its backup correctly.

Testing your database backups

An often-overlooked step is to test the backups you have created. Someday you’ll need one of your backup files and you won’t want to open it up to find an empty file!

One way to test your backup is to create a new database, import the backup file in to it and perform a query or two to make sure some recent data exists in the backup so that you know your recent back has worked.

To create a temporary database called temp_database use the following command:

mysql –host=${DATABASE_HOST} –user=${DATABASE_USERNAME} –password=”${DATABASE_PASSWORD}” -e “CREATE DATABASE IF NOT EXISTS temp_database”;

Next, import your backup into the database. You’ll need to decompress it first if you compressed it using the following commands:

gzip -d backup.sql.gz
mysql –host=${DATABASE_HOST} –user=${DATABASE_USERNAME} –password=”${DATABASE_PASSWORD}” temp_database < backup.sql

This may take a moment or two to complete depending on the size of your backup

Once the database has been restored, the next step is to query one or more tables to make sure it has recent data. The table you query really depends on your own database and the information you store there.

One potential example is to query a table called ‘logins’, if you happen to have an application that makes a record of each time a user logs in.

Assuming the table has a date/time field of some kind, you could run a query to check to see if at least 1 record exists in the logins table from yesterday or some period of time that makes sense to you as shown in the following snippet:

mysql –host=$HOST} \
–database=${DATABASE} \
–user=${local_db_user} \
–password=${local_db_password} \
–skip-column-names
-e “select count(*) from logins where date > 2020-01-01” > logins_count.txt

This command will connect to the database and query the logins table to count the number of logins after 1st January 2020. It will save the result to a text file called logins_count.txt

The skip-column-names parameter will prevent the text file from containing the field name from the table, making it a little easier to confirm that the file contains only a number and that it is greater than 0, when performing a check with our code.

If the text file exists and has a count greater than 0 then your earlier backup appears to have backed up correctly.

In our next section, we will look at the back up facility built in to AWS’s Relational Database Service (RDS). We will look at how they differ from manual backups and how to enable them.

Using RDS Snapshots

AWS Relational Database Service (RDS) is one of many services provided by AWS. At a basic level RDS is a database service which can be used to create relational databases using a number of providers such as mySQL, MariaDB, Oralceor Aurora. Aurora is a mysql compatible relational database created by AWS.

RDS is tempting to use when you need a database for your application. It can handle database replication, backups and restores for you which can be difficult to develop and maintain if you are self-hosting a database on one or more EC2 instances.

RDS provides a backup functionality built in. This backup process allows you to choose a retention period of 0 to 35 days. This means RDS will back up your database each day and keep those backups for up to 35 days or whichever number of days you choose. If your RDS instance ever ran into difficulty, such as losing data due to an incorrect delete, you can ‘Revert’ to an earlier version of the database within the retention period. In this context, ‘Revert’ may be a little bit misleading for this process, as it will create a new RDS instance from the backup, leaving your original RDS instance in place.

You can also choose a time period when these backups are performed during a day. You can choose a start time, such as a quiet time overnight when your database isn’t too busy and make a period of time such as an hour available for performing the backup. While the backup is taking place, your RDS instance can operate as normal.

RDS Snapshots are visible in the Snapshots section of RDS. You have a number of available options when working with Snapshots including:

  • You can Restore the snapshot, meaning you can create a new RDS instance from the backup.
  • You can Copy the snapshot, meaning you can copy the snapshot to another AWS region, such as from us-east-1 to eu-west-1.
  • The Share Snapshot option allows you to make the snapshot Public if you choose to do so, or to make it available to another AWS Account ID.
  • You can also export your snapshot to S3. S3 is AWS’s storage service and the export option allows you to export all or some of the backup.

In our next section, we will look at a relatively new backup service provided within AWS called ‘AWS Backup’. AWS backup can help to back up a number of different items within AWS and also help to manage their lifecycle, meaning how the backup files are handled over time.

Using AWS Backup service

In 2019, AWS launched a new backup service called AWS Backup. This service allows you to automate some of the existing backup options that already exist within some other AWS services such as RDS.

AWS Backup starts by creating a Backup Plan. Within this plan you can define a schedule such as daily, weekly, monthly or a custom Cron expression for more fine grained control and you can define a backup window for when the backups should take place.

A Lifecycle section allows you to transfer backed up files to cold storage. This is a storage option that provides a lower cost to store your data which happens automatically after a defined period of time such as 1 week after the backup was created. You can take an additional step to Expire this backup after a period of time.

As an example of putting these lifecycle options together, an item can be backed up on the first day of the month and transitioned to cold storage the following day. At the end of the month or even the end of the year it could be deleted automatically.

You can create different backup Vaults as a way to use different Key Management Service (KMS) encryption keys for your backups. This can be useful if you work with different customers and want to backup and encrypt backups for that particular customer using their own dedicated KMS key.

Like many other AWS services you can add some Tags to your backup plan. This can be useful later for reporting or cost analysis purposes.

Once your Backup plan is created, you can then move on to the step of defining the items to back up. This section is called Resource Assignment. You can give your resource assignment a name and attach an Identity and Access Management (IAM) role which has the permission to perform the backup of the assigned item(s).

The final step is to assign the resources you would like to back up. You can assign resources based on Tagged items, or specific resource IDs such as RDS instances.

You can combine several items to back up. You could back up 1 RDS instance and 1 EC2 instance as part of 1 backup plan and have them both transition to cold storage and expire after a period of time.

Overall AWS Backup is a very useful service to use. You can spend some initial time setting up one or more backup plans and then leave it to work automatically behind the scenes with no additional work so that copies of your databases and EC2 instances are available if you ever need them.

Following on from our discussion in an earlier post IAC tools such as Terraform or Cloudformation can be used to create AWS backup plans, so you could create the plans while developing your infrastructure too.

A number of backup options would be useless without AWS S3, so in our next section we will look at S3 and the options it contains to help store and manage our back up files.

Using AWS S3 for storing backups

AWS S3 is an excellent resource for easy and accessible storage. You don’t need to allocate any storage in advance, it can grow along with your usage.

If you are developing a web application and users of your application can upload files such as logos or avatar images, storing those items directly to S3 is an ideal solution. When using s3 you need to create one or more “buckets” to start storing your files.

Bucket names need to be unique across all AWS customers, so common bucket names such as “files” are probably already taken. Instead, consider creating a bucket with a name such as {your application name}-files for a better chance of creating a relevant available bucket.

When creating or modifying an existing S3 bucket, you will have a number of options. AWS have added additional options and storage types to S3 over time, but the following are currently the most common and useful options:

  • Versioning – With Versioning enabled on an S3 bucket, any file that is uploaded to the bucket can be overwritten and earlier versions of the files will be retained. This can be used as a form of backup and versioning needs to be enabled for the feature called Replication.
  • Replication – Replication allows all or some items in a bucket to be immediately copied to another s3 bucket. When an object is copied, it can change to a different storage class. This means that an object can be placed in your S3 bucket and immediately copied to another s3 bucket with a different storage type of ‘Glacier’ for lower cost.
  • Lifecycles – Just like the AWS backup service described earlier, a lifecycle can be applied to an s3 bucket. A lifecycle rule can be used to change the storage type of objects or delete the file after a period of time.
  • Storage Types – S3 provides a number of different storage types. S3 Standard (S3 Standard), Amazon S3 Intelligent-Tiering (S3 Intelligent-Tiering), Amazon S3 Standard-Infrequent Access (S3 Standard-IA), Amazon S3 One Zone-Infrequent Access (S3 One Zone-IA), Amazon S3 Glacier (S3 Glacier). Each tier has different advantages such as lower cost, though comes with a down side too such as reduced availability. If you are unsure, use the S3 standard storage option.

AWS provides an S3 FAQ here for any further questions you might have https://aws.amazon.com/s3/faqs/

Now that we know about S3 buckets and how to enable options such as a lifecycle, in our next section we will look at ways to get our backups uploaded to S3.

Backing up your files to S3

In an earlier post in a section called ‘Installing the AWS CLI‘, we described using the AWS CLI to create resources on AWS such as EC2 instances.

We can also use the AWS CLI to create and manage S3 buckets.

To create a new S3 bucket from your terminal, using the AWS CLI use the following command:

aws s3api create-bucket –bucket myapplication-backup-bucket

To list your S3 buckets you can use:

aws s3 ls

To list the contents of an S3 bucket use:

aws s3 ls s3://{your bucket name}

The above image shows the output from creating a bucket and querying its objects

To copy a file from your local machine to an S3 bucket use:

aws s3 cp file.txt s3://{my bucket name}/file.txt

In our next section we put some of our AWS CLI knowledge to good use. We will write a script to back up a database and upload the resulting back up file to AWS S3 for storage.

Database backup with Mysqldump and upload to S3

Putting the items we have learned above together, we can create a script that can connect to a database, perform a backup, compress the backup file and upload it to s3. Backup.sh

#!/usr/bin/env bash
# The following assumes you have an AWS account with the AWS CLI installed locally
# It will perform a mysqldump of a database, compress and upload the resulting file

DATABASE_HOST=”localhost”
DATABASE_USERNAME=”user”
DATABASE_PASSWORD=”password”
DATABASE=”myapplication”
S3_BUCKET=”myapplication-backup-bucket”

# dump the database to a local file
mysqldump –host=${DATABASE_HOST} –user=${DATABASE_USERNAME} –password=”${DATABASE_PASSWORD}” ${DATABASE} –single-transaction > ${DATABASE}.sql

# compress the sql file using gzip
gzip ${DATABASE}.sql

# upload the compressed database file to s3
aws s3 cp ${DATABASE}.sql.gz s3://${S3_BUCKET}/${DATABASE}.sql.gz

# list contents of the s3 bucket to confirm the upload
aws s3 ls s3://${S3_BUCKET}

# remove the local compressed database file
rm ${DATABASE}.sql.gz

Once your database is backed up it is important to test that the backup was successful. We will create two scripts to validate our backup. The first script will download the database backup and use it to create a local database. The second script will query the local database to make sure it has some data, to confirm it was backed up correctly.

In your editor, create a new file called prepare.sh with the following content:

#!/usr/bin/env bash
# The following assumes you have an AWS account with the AWS CLI installed locally
# It will download a backup file, create a database, import your data and perform a query that should have at least 1 resulting record.

DATABASE_HOST=”localhost”
DATABASE_USERNAME=”user”
DATABASE_PASSWORD=”password”
DATABASE=”myapplication”
S3_BUCKET=”myapplication-backup-bucket”

# download a backup file from s3
aws s3 cp s3://${S3_BUCKET}/${DATABASE}.sql.gz ${DATABASE}.sql.gz

# uncompress the backup file
gzip -d ${DATABASE}.sql.gz

# create a local database called ‘validate’
mysql –host=${DATABASE_HOST} –user=${DATABASE_USERNAME} –password=”${DATABASE_PASSWORD}” -e “create database if not exists validate;”

# import the backup in to the database
mysql –host=${DATABASE_HOST} –user=${DATABASE_USERNAME} –password=”${DATABASE_PASSWORD}” ${DATABASE} < ${DATABASE}.sql

The above script will connect to AWS S3 to download our backup file. It will decompress the file and create a local database using the content from the backup.

You may need to change the variables at the beginning of the file to match your database settings so the script can connect successfully.

Our next step is to query some of the data in the local database to verify that the data was backed up correctly.

In your editor, create a new file called validate.sh with the following content:

#!/usr/bin/env bash
# The following assumes you have a local database already created to query
# It will perform a query that should have at least 1 resulting record.

DATABASE_HOST=”localhost”
DATABASE_USERNAME=”user”
DATABASE_PASSWORD=”password”
DATABASE=”myapplication”

# query the data and store the result in to a file
mysql –host=${DATABASE_HOST} –database=${DATABASE_USERNAME} –user=${DATABASE_USERNAME} –password=${DATABASE_PASSWORD} –database=${DATABASE} –skip-column-names -e “select count(*) from users” > users_count.txt

# count the number of records in the file created by the previous query
record_count=$(cat users_count.txt)

# if there are more than 0 records, the backup file is OK
if [ ${record_count} -gt 0 ]; then
echo “Database backed up successfully”
else
echo “No records, the database didn’t back up properly”
fi

# clean up local files
rm users_count.txt
rm myapplication.sql

# drop the database
mysql –host=${DATABASE_HOST} –user=${DATABASE_USERNAME} –password=”${DATABASE_PASSWORD}” -e “drop database if exists validate;”

Now that we have created our files, we need to run them and confirm the results by following these steps:

In your terminal run the following command to run the first script:

bash prepare.sh

The script will download the backup file from S3 and create a local database ready to query.

Next run the second script to query the data:

bash validate.sh

If your data backed up correctly and there is data in your users table, then you should see an output of:

Database backed up successfully.

If there is no data in the table, or some other issue with your database then you will see an output of:

No records, the database didn’t back up properly.

If you receive this error, refollow the steps from earlier in this post to make sure you have backed up your data correctly so that the validation process will run successfully.

Database backup and upload to S3 with Ansible

To use ansible to back up our database and to upload the file to s3, we will create 2 small Ansible roles:

  • The first role will connect to the database and dump it to a local file and compress it using gzip.
  • The second role will create an s3 bucket and upload the compressed backup file.

In the following steps we will create the files necessary for each role. Before you begin, make sure you have an AWS Access Key and AWS secret key ready from the IAM section of your AWS Account. 1. The first step is to create an Ansible Vault file to store AWS credentials for Ansible to have permission to create an s3 bucket and upload a file to it:

ansible-vault create group_vars/all/pass.yml

Once the file is opened, add the following content to the file:

ec2_access_key: [your aws access key ]
ec2_secret_key: [your aws secret key]

You will be asked to provide a password. Remember this password as you will need it any time you run the playbook.

Next, create backup.yml with the following content, you can change the database variables to suit your own database details:


– hosts: localhost
connection: local
gather_facts: no

vars:
database: myapplication
database_username: user
database_password: password
database_host: localhost
name: Backup a database

roles:
– backup

The next file we will create is the backup role. Create a file at the location roles/backup/tasks/main.yml with the following content:

– name: Dump database
mysql_db:
state: dump
login_host: “{{ database_host }}”
login_user: “{{ database_username }}”
login_password: “{{ database_password }}”

name:
– “{{ database }}”

target: “{{ database }}.sql”
– name: Compress database backup

archive:
path: “{{ database }}.sql”
dest: “{{ database }}.sql.gz”
format: gz

For the s3 playbook, create the following file s3.yml with the following content:


– hosts: localhost
connection: local
gather_facts: no

vars:
bucket: myapplication-backup-bucket-ansible
file: myapplication.sql.gz
name: Backup and upload to s3

roles:
– s3

Create the s3 role file at roles/s3/tasks/main.yml with the following content:

– name: Create an s3 bucket

aws_s3:
bucket: “{{ bucket }}”
mode: create
– name: Upload backup file to s3
aws_s3:
bucket: “{{ bucket }}”
object: “{{ file }}”
src: “{{ file }}”
mode: put

Finally, create one last file to include a list of dependencies that Ansible will need to enable it to perform the dump of the database, requirements.txt

pymysql

Use the following command to make sure the dependencies are installed for Ansible to work:

pip install -r requirements

Next run the backup playbook to backup the database and compress the file:

ansible-playbook s3.yml –ask-vault-pass

Then run the s3 playbook to create an s3 bucket and upload the database backup to s3:

ansible-playbook backup.yml –ask-vault-pass

The result is that Ansible will back up your database, compress the output file, create an s3 bucket and upload the compressed backup file to your new s3 bucket.

The source code for using Ansible to back up and upload the resulting file to S3 is available here https://github.com/gordonmurray/mysqldump_s3_ansible

In this post we covered the importance of backing up a database as well as validating that the backup was successful, by querying a table for recent data. We learned about the Snapshots option that can be used with AWS Relational Database Service (RDS) and how AWS S3 is an ideal location to store backed up data We worked to combine the steps that we have learned in to a number of useful scripts to help automate the process of backing up and validating a database. We looked at the AWS Backup server to help back up Tagged items in our AWS account so that they can be backed up automatically. To expand on our existing knowledge of Ansible from earlier posts we also learned how to perform backups using an Ansible playbook.

 

 

 

 

An introduction to performing code, infrastructure or database deployments

In this post we will look at some of the approaches we can use to deploy code or infrastructure changes into a production environment. Regardless of the programming language you may be using to develop a web site or application, from HTML/CSS, NodeJS, Java or Go, there are likely some build steps you need to take to make your application ready for production. Build tools can help you to perform those steps quickly and automatically, saving you time in repeating the same steps any time you want to deploy even a small change.

In a previous post, we were introduced to CloudFormation, Terraform, Ansible and Pulumi. In this post we will look at ways we can deploy those changes into production. While these examples are specific to performing infrastructure changes, the same approach can and often is applied to deploying code changes into production. In our code examples we will include code changes along with infrastructure changes where possible.

Some of the tools we will cover in this post can be run directly from a terminal on your local PC or laptop. This can work well if you are the only developer working on a project. If you are working as part of a team however, it is best to use a central or dedicated system for deploying changes so that any changes can be visible to your other team members. In this post, we will introduce some possible approaches that can be used to build or deploy changes to a web application or a database to production use. We will start with using the command line for deployments and then use Git pull to deploy code changes. We will also learn to use third party tools or services to deploy changes to production and learn to deploy a simple webserver using Travis. Towards the end of the post we will learn to replace a webserver using Packer, Ansible and Terraform and also learn to use Skeema to create or alter database tables.

In this post, we will cover the following topics:

  • Using the Command line for deployments
  • Using Git pull to deploy code changes
  • Using third party tools or services to deploy changes to production
  • Deploy a simple webserver using Travis
  • Replacing a webserver using Packer, Ansible and Terraform
  • Using Skeema to create or alter database tables

Using the Command line for deployments

Issuing commands from your local machine or combining commands into a bash script isn’t the best process for deploying web application changes to production.

At times however, performing a deployment or rolling back a deployment to an earlier version from the command line can be the fastest approach to get some changes made to production in an emergency scenario.

When compared to logging in to a cloud environment such as AWS to perform some actions, using the AWS CLI can save a lot of time.

Incidents do happen and if services such as GitHub are experiencing down time, you might still need to deploy your code into production, this is where command line steps can help.

Using Git pull to deploy code changes

If you are using services such as Github.com, it’s a good idea to get to know some of the most common Git commands such as git clone, git commit, git push and git pull, so that we can run those commands on remote servers or through our build tools.

There are many more Git commands and fortunately there are desktop clients such GitHub Desktop (https://desktop.github.com/) to help you manage your files without knowing every command.

Instead of using commands such as FTP or SCP to copy files from your local machine to the destination server, one of the options with Git is to perform a git pull. This is the process of connecting to a remote server, installing Git to clone the project from GitHub or Bitbucket and then perform a git pull to bring any new changes to the code onto the server.

This process will work however, it also has several disadvantages. If you have many servers, you will need to connect to each server and perform a git pull on each server. This is time consuming and can cause conflicts if you or your team routinely SSH into servers to change files directly as the files in GitHub will differ from the files on the server.

Using Git in this way is also a security concern. Cloning the project on to a server using git will leave a .git folder on the server. This folder is essential for Git to work locally though it also contains the history of all file changes of your project. If you ever committed database passwords or API keys to your code, a third party may be able to see those details in your git history. This folder can also take up a lot of disk space on your server, depending on the size of your project.

Using third party tools or services to deploy changes to production

When working on small or solo projects, it can be convenient to deploy changes from your own local machine to your production servers. As a project grows, or you begin working with others, having a central place such as Github to store your code is essential and having a dedicated service to deploy changes can significantly improve your development workflow.

When using services such as Github or Bitbucket your code is stored in a central place called a Repository. You can continue to develop locally on your own machine and commit changes often. Once you have added your changes, the code is available to coworkers to review or merge into the main project.

As well as working with others on a project, using a central repository can trigger a number of other useful processes. Popular third party services such as Travis can watch for changes to your repository and then trigger some build steps to test your code or deploy it to production.

In this post we will try a few different methods of deploying code to production servers so you can gain a little experience with each one, to see if one of them works well for your own projects. Many 3rd parties such as https://circleci.com/ or https://travis-ci.com/ exist to help out with software deployments.

Services such as these can connect to your Github or Bitbucket repository and then then deploy the files that have changed to your target server(s).

These services can be great to work with. When you commit some code changes to your repository, the service can almost immediately push those changes out to your servers.

They can also provide follow up steps, such as sending you and your team an email letting everyone know that the latest work has now been deployed on the servers.

If you deploy some new code that happens to have a bug, many services also provide a great interface to roll back your changes to the previous version of your work quickly.

Depending on your software of where you are deploying to, these services can often run pre and post deployment commands on your server. For example, you might need to stop a currently running application, deploy your latest changes and then start up or restart the application afterwards. These services can perform the steps for you every time, so you don’t have to. You can focus on writing your software and not worry about missing any routine build or restart commands when deploying any changes, as a good build tool will perform those actions for you.

You could also use those pre deployment hooks to back up your existing files or database first as a precaution before deploying changes to your system. We will cover ways to back up your files and databases in the next post.

There is often some up-front time required to set up these third party services to deploy your work but once set up, they can work very well in the background for you to build and deploy your projects.

If you need to go beyond the options supported by 3rd party tools, such as an unsupported programming language, or needing to self-host your build tools, then tools such as Travis or Jenkins might be a better fit.

Travis allows you to run a number of builds in parallel, for example testing your code on different OS versions or different programming language versions. Travis also allows you to create a .travis.yml file so that you can create your build steps and commit those changes to your project repository next to your application and infrastructure code.

Jenkins can be a great solution if you want to self-host your build tools in your own environment for compliance reasons. Jenkins supports a Jenkinsfile that you can commit to your source code repository just like Travis.

In our next section, we will take some existing work from an earlier post and deploy the same code using Travis.

Deploy a simple webserver using Travis

In this section, we will take our previous work of building and configuring a webserver using Ansible and connect it up with https://travis-ci.org. If you don’t have an account with https://travis-ci.org/ take a moment to create one now. There is no cost for travis-ci.org if you are developing an open source project. Alternatively, you can also use https://travis-ci.com/ as this will work with private repositories on Github.

You can describe the steps you would like Travis to take by creating a .travis.yml file. Travis also uses the YAML format for its build steps.

In this section we will create a new .travis.yml file and add the necessary steps to install Ansible and run the same commands we would run locally to run the Ansible playbook.

We will add this file to the root folder of our Github repository. When we make any changes to our code in the repository, Travis will be notified and start a build process following the steps we have in our.travis.yml file.

In the same folder where you created your Ansible playbook from our last post, create a .travis.yml file. In this file we will create a number of stages to install and deploy our changes. There are more Stages available for different functionality and for the full information on those stages view the Travis website at:

First we will use the os, dist, language and python keys to tell Travis the environment we want to create to run our build. In this example, we want to use a Linux instance using Ubuntu Bionic and use Python version 3.6: 1. Create a new file called .travis.yml and begin by adding the following lines:

os: linux
dist: bionic
language: python
python: 3.6

Next we will use the before_install key to perform an apt update to make sure we have an up to date instance for the next install step. We add the -qq parameter to make sure the output is minimal so the screen in the Travis UI is not filled up with apt update steps.

We then write the VAULT_PASSWORD environment variable to a local file for Ansible to use in a later step when running the playbook:

before_install:
# Scripts to run before the install stage
– sudo apt update -qq
– echo “$VAULT_PASSWORD” > vault_password

In this step we use the install key and use a Python package manager called pip to install awscli, boto3 and ansible version 2.9. Ansible uses the AWS CLI and boto3 to communicate with AWS:

install:
# Scripts to run at the install stage
– pip install awscli boto boto3 ‘ansible==2.9’

Next we form the deploy step. We will use 2 deploy steps. The first step will call our infrastructure playbook to make sure our infrastructure items are in place. The next step will run the webserver playbook to configure our server and deploy our files.

aws_ec2.yml is used to query the AWS resources to find the EC2 instance(s) by Tag so it knows which instance(s) to configure. Each playbook uses the vault password file to decrypt the sensitive credentials.

We use the on key to tell Travis to only deploy this work if it is triggered by a commit to the master branch on Git. This allows for commits to other branches or PRs without triggering a build as shown in the following code block:

deploy:
# deploy infrastructure changes
– provider: script
script: ansible-playbook -i aws_ec2.yml infrastructure.yml –vault-password-file=vault_password
on:
branch:master

# deploy webserver configuration
– provider: script
script: ansible-playbook -i aws_ec2.yml webserver.yml –vault-password-file=vault_password
on:
Branch:master

Finally, we add the after_script step to run as the last stage and to delete the vault_password file just to be just

after_script:
# Scripts to run as the last stage
– rm -f vault_password

Travis provides both online and command line option to validate your .travis.yml file, info on both options can be found at https://support.travis-ci.com/hc/en-us/articles/115002904174-Validating-travis-yml-files

Once you are happy with your .travis.yml file, you can commit the files to the repository to trigger a Travis build using the following steps in your command line:

git add .travis.yml
git commit -m “Adding Travis file”
git remote add origin git@github.com:{YOUR_GITHUB_USERNAME}/ansible-webserver.git
git push -u origin master

Your code will now be pushed to Github. Open your Travis dashboard at https://travis-ci.org/{YOUR_TRAVIS_USERNAME}/ansible_webserver and you should see your project build, performing the same steps you performed locally to create the EC2 instance and configure the server using Ansible.

You now have a Github repository that stores your project code. Any time you’d like to work on the project you can clone or pull the project, make some changes and then commit and push the changes back to the repository to trigger a build to further improve your infrastructure or webserver.

The above code is available here: https://github.com/gordonmurray/learn_sysops/tree/master/chapter_5/ansible

Git and Github are capable of many more items, such as branching or forking code to allow independent development when working as part of a team, though a guide on how to master using Github is outside the scope of this book.

Github have a Github Guides site you can follow here to learn more https://guides.github.com/

In this section, we used Travis to deploy changes to an existing server. In our next section we will use a combination of tools including Packer to create a new server image. We will use Ansible to configure this image and then we will use Terraform to put this new image into place as a working server to replace an existing server.

Replacing a webserver using Packer, Ansible and Terraform

In our earlier post, we used Ansible to create an EC2 instance first, and then took steps to configure it afterwards. In this process, the EC2 instance is usually built once then updated from then on with configuration or code changes.

In this section we will take a different approach. Instead of updating an existing server, we will create a new server, configure it and deploy it. We will use a tool called Packer from Hashicorp, the same company that behind Terraform.

To install Packer, please follow the instructions for your system here from the Packer website: https://packer.io/intro/getting-started/install.html

Servers created in this way are often called ephemeral or temporary servers. Creating servers in this way can help with auto scaling servers during high and low traffic. A pre-made image can be used to increase the number of active servers to handle peak web traffic and scale down off peak to save money.

In a larger organization, this process of replacing a server can help with security measures and upgrade scheduling, since a server image can be created, configured and tested every time there is a code change and deployed into production at any time.

Packer is a tool to create Amazon Machine Images (AMIs) that AWS can understand and use to create EC2 instances. We will use Packer to start with an Ubuntu image and use Ansible to configure the server.

Once we have created an AMI, Terraform can be used to create an EC2 instance using the AMI created by Packer. Since Packer and Ansible will have configured the server already, then when Terraform creates the EC2 instance it will already be ready to serve its website with no further changes.

We will combine some Ansible and some Terraform from our earlier post. Create a new folder called packer-terraform-webserver and we will create the following files:

  • /packer/server.json
  • /packer/variables.json
  • /ansible/

Copy your Ansible files from the last post to this folder

  • /terraform

Copy your Terraform files from the previous post to this folder and we will use an updated ec2.tf file:

  • /ansible/server.yml
  • /packer/server.json

The first file we will create is server.json. It includes a number of parts, so let’s look at each section as we append to the file:

Create a new file located at /packer/server.json with the following content:

{
“builders”: [{
“type”: “amazon-ebs”,
“profile”: “{{user `profile`}}”,
“region”: “{{user `region`}}”,
“source_ami”: “{{user `base_ami_id`}}”,
“instance_type”: “{{user `instance_type`}}”,
“force_deregister”: “true”,
“force_delete_snapshot”: “true”,
“ssh_username”: “ubuntu”,
“ami_name”: “example”,
“ami_regions” : [“{{user `region`}}”],
“tags”:
{
“Name”: “webserver”
}
}],

This builders section provides the main information for Packer to know the kind of AWS AMI to create. It also uses a number of vaiable values that we will define in another file later.

Next, add the following section to the same file:

“provisioners”: [
{
“type”: “shell”,
“inline”: [
“sudo apt update”,
“sudo apt install python3-pip -y”,
“pip install ‘ansible==2.9′”
]
},

This is a shell provisioner. It connects to the AMI that Packer has started and runs some necessary installation steps for the proceed steps to operate normally.

Next add the following to the same file:

{
“type”: “ansible-local”,
“playbook_file”: “../ansible/server.yml”,
“role_paths”: [
“../ansible/roles/apache”,
“../ansible/roles/php”,
“../ansible/roles/mysql”,
“../ansible/roles/deploy”
],
“group_vars”: “../ansible/group_vars”
},

This is an Ansible provisioner. It lets Packer know that we want to use Ansible to configure a server and instructs Packer where to find our Ansible playbook and our Ansible roles.

Finally, in the same file add the following section to upload some files

{
“type”: “file”,
“source”: “../ansible/roles/apache/files/webserver.conf”,
“destination”: “/home/ubuntu/webserver.conf”
},
{
“type”: “shell”,
“inline”: [
“sudo mv /home/ubuntu/webserver.conf /etc/apache2/sites-available/webserver.conf”
]
},
{
“type”: “file”,
“source”: “../src/index.php”,
“destination”: “/var/www/html/”
}
]
}

This section uses a file and shell command to upload our conf files and a shell command to move the conf file into the correct location.

Next, we will create the other files we will need. These files are shorter so we can take each one at a time instead of breaking it down in to several steps per file.

/packer/variables.json

{
“base_ami_id”: “ami-04c58523038d79132”,
“profile”: “example”,
“region”: “eu-west-1”
“instance_type”: “t2.nano”
}

/terraform/ec2.tf

# Get AMI
data “aws_ami” “webserver_ami” {
most_recent = true

filter {
name = “name”
values = [“webserver*”]
}

owners = [“${var.aws_account_id}”]
}

# Create EC2 instances
resource “aws_instance” “webserver” {
ami = “${data.aws_ami.webserver_ami.id}”
instance_type = “${var.default_instance_type}”
vpc_security_group_ids = [“${aws_security_group.example.id}”]
subnet_id = “${aws_subnet.subnet-1a.id}”
key_name = “${aws_key_pair.pem-key.key_name}”

tags = {
Name = “webserver”
}

}

You can validate Packer files using:

packer validate -var-file=packer/variables.json packer/server.json

Once validated, you can build the AMI with Packer using:

packer build -var-file=packer/variables.json packer/server.json

This will take a few minutes to complete. This will connect to your AWS account, start a small EC2 instance and Ansible will then configure the image.

Once it has completed, it will create an AMI in your account and terminate the EC2 instance it was using.

This AMI is then available for Terraform to find and use. The ec2.tf file has the aws_ami to find it based on its name and then aws_instance uses it to create the image.

To deploy the image with Terraform use:

cd /terraform
terraform init
terraform apply

Terraform will show you the items it plans to build so you can review before continuing. If you would like to go ahead, enter Yes and Terraform will create the EC2 instance.

If you would like to clean up your AWS account and remove the EC2 instance and surrounding items such as Security groups and key pairs, you can use the following command:

terraform destroy

This command will ask if you are sure that you want to continue before it removes the items from your AWS account.

You now have a process whereby Packer can create a base image, using Ansible to configure anything you need on the server and then Terraform to deploy the server.

If you make changes to Ansible so that your server is configured differently, for example if you change from Apache to Nginx, or deploy another website, the Packer will create a new AMI which replaces your last AMI. Terraform will then terminate your currently running EC2 instance and put a newer version in its place, within the same VPC, subnet and security group settings as the previous instance.

This process of replacing a server instead of updating it is known as ephemeral infrastructure. When a server is removed, you will lose any data from that server. This may seem scary and dangerous, but if we are backing up our data and abstracting our databases to services such as AWS Relational Database Service (RDS) and our storage to services like S3, we can be sure our data will remain as servers change over time. We will cover backups, testing and restoring data in a future post.

To remove the items created by Terraform, use the following command:

terraform destroy

To delete the AWS IAM user created for Ansible and Terraform, use the following command:

aws iam delete-user –user-name example

To identify the AMI created by Packer, get the AMI ID value using the following command:

aws ec2 describe-images –filters “Name=tag:Name,Values=webserver” –profile=webserver –region=eu-west-1 –query ‘Images[*].{ID:ImageId}’

Then delete the AMI, use the following command:

aws ec2 deregister-image –image-id ami-<value>

The source code for Ansible, Packer and Terraform and the steps mentioned above are all available here: https://github.com/gordonmurray/learn_sysops/tree/master/chapter_5/packer_ansible_terraform

Using Skeema to create or alter database tables

Until now we have looked at code related or infrastructure related changes. Database changes are also a common aspect of software development.

In this section we will use Skeema to create and alter some tables in a MariaDB database.

Skeema is a very useful tool that works with mySQL or Mariadb databases to help you to compare your development database files to a production database. Skeema will generate any alter statements that are required and apply those changes to your production database.

Skeema is a free and open source tool developed using Go and is available on Github at https://github.com/skeema/skeema

If you have very large tables in your production database, performing alters to the tables can take a lot of time to complete fully and may cause problems for your users if there are users actively using your software while you change the database.

Skeema has an option to use an external service to alter your tables. In the example that follows we will get up and running with Skeema and in another example we will update Skeema to use Perconas pt-online-schema-change to alter a pretend larger table.

The Percona alter can perform the necessary work by copying the target table structure to a new temporary table, then copying any data from the existing data to the new temporary table. It can then rename the tables to that new temporary table becomes the main table and optionally drop or leave the old table in place.

Skeema won’t let you make dangerous changes such as dropping a column. Sometimes however you might need to do this kind of change so you can tell Skeema to continue by using the –allow-unsafe parameter when performing a Push to change to a production database.

The following steps assume you have access to a mySQL or MariaDB database to try Skeema. If you don’t already have a database installed, you can follow the installation instructions here on the MariaDB website https://mariadb.com/kb/en/getting-installing-and-upgrading-mariadb/

If you are on a Linux based machine, you can install MariaDB locally by using the command:

sudo apt install mariadb-server

To show Skeema in operation, create a new database called prodution using the following command:

CREATE DATABASE IF NOT EXISTS production;

If you would like to create a dedicated database user to use with Skeema, create a database user using the following command:

grant all privileges on *.* to ‘dbuser’ @’localhost’ identified by ‘password’;

You can change the username of dbuser and its password instead of password to whatever you wish to use.

In the prodution database, lets create a simple users table with some data to mimic a production database.

Use the following to create a users table and insert a couple of records:

use production;

CREATE TABLE `users` (
`id` int(11) UNSIGNED NOT NULL AUTO_INCREMENT,
`first_name` varchar(100) NOT NULL,
`last_name` varchar(100) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4;

INSERT INTO `users` (`first_name`, `last_name`) VALUES
(‘John’, ‘Murphy’),
(‘Fred’, ‘Smith’);

Now that we have an example production database, with a table, some data and a database user, next we can install Skeema to start making changes to that database.

The easiest way to use Skeema is to get a compiled version of it from the Releases page on Github, suitable for your system at https://github.com/skeema/skeema/releases

Once you have Skeema installed, ensure it is working by running the following command:

./skeema version

You should see an output similar to “skeema version 1.4”.

So that we can keep things organized lets create a folder called database to contain our work:

mkdir database && cd database

We can initialize Skeema by using the following command:

./skeema init -h 127.0.0.1 -u dbuser -ppassword -d development

In this command we told Skeema to initialize and told it that our database is accessible at 127.0.0.1, that we are using the database user called dbuser and a password of “password” and finally to create a folder to represent our database called development

If you have existing databases and tables present in your database, Skeema will create a folder to represent each database and place .sql files in to each one to represent the table schemas.

Since we created a database called production and added a table called users you should now see a folder called production and within it a file called users.sql. You will also notice a file called .skeema in the folder which contains some configuration information for Skeema.

From here, we can add or alter any .sql files in the Production folder. As you develop a software application you will probably change your database requirements over time. You might use a tool such as PHPMyAdmin or Navicat as an interface to your database to help you to make changes to your local database. You can use those tools to export your tables into the production folder and Skeema will do the work of determining the changes to make and apply them to your production database.

To simulate a change, lets create a new file called production/comments.sql with the following content:

CREATE TABLE `comments` (
`commentid` int(10) UNSIGNED NOT NULL AUTO_INCREMENT,
`body` text NOT NULL,
`created` DATETIME NOT NULL,
`modified` DATETIME NOT NULL,
PRIMARY KEY (`commentid`)
) ENGINE = INNODB DEFAULT CHARSET = utf8mb4;

We could run this SQL file directly on the database but instead we will use Skeema to compare our production folder to our database and apply any changes.

To get Skeema to show us the changes it will make, run the following command:

./skeema diff -ppassword

You should see an output similar to the following, telling you that Skeema wants to create the new comments table:

[INFO] Generating diff of 127.0.0.1:3306 development vs /home/database/production/*.sql
— instance: 127.0.0.1:3306
USE `production`;
CREATE TABLE `comments` (
`commentid` int(10) unsigned NOT NULL AUTO_INCREMENT,
`body` text NOT NULL,
`created` datetime NOT NULL,
`modified` datetime NOT NULL,
PRIMARY KEY (`commentid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
2020-04-09 15:50:43 [INFO] 127.0.0.1:3306 production: diff complete

You can get Skeema to apply this change by running the following command:

./skeema push -ppassword

You should see an output similar to the following:

[INFO] 127.0.0.1:3306 production: push complete
[INFO] Pushing changes from /home/database/production/*.sql to 127.0.0.1:3306 production
[INFO] 127.0.0.1:3306 production: No differences found

If you connect to your production database, you will need the new Comments table has been created there, ready for your application to use.

If you or your coworkers have altered the production database directly, you can tell Skeema to re-read in the details from the production database by running the following command:

./skeema pull -ppassword

If you created or changed any tables, the files in production/*.sql will be created or updated to match.

Using Skeema with Percona’s pt-online-schema-change

Performing alters to large database tables can take a lot of time to complete fully and may cause problems for your users if there are users actively using your software while you change the database.

The Percona pt-online-schema-change tool can perform the necessary work by copying the target table structure to a new temporary table, then copying any data from the existing data to the new temporary table. It can then rename the tables to that new temporary table becomes the main table and optionally drop or leave the old table in place. You can read more about pt-online-schema-change here : https://www.percona.com/doc/percona-toolkit/3.0/pt-online-schema-change.html. Skeema can be updated to use pt-online-schema-change very easily.

The first step is to install the pt-online-schema-change tool locally. The schema change tool is part of the Percona Toolkit and can be installed on a Linux machine using the command:

sudo apt install percona-toolkit

Other Linux based installation details can be found at https://www.percona.com/doc/percona-toolkit/LATEST/installation.html

To ensure pt-online-schema-change is installed correctly, use the following command:

pt-online-schema-change –help

You should see a large output of usage parameters that the tool supports.

To update Skeema to use pt-online-schema-change, open the file at /production/.skeema and add the following:

alter-wrapper=/usr/bin/pt-online-schema-change –execute –alter {CLAUSES} D={SCHEMA},t={TABLE},h={HOST},P={PORT},u={USER},p={PASSWORDX}

If you run the Skeema diff command again, you will notice a slightly longer output that include pt-online-schema-change:

./skeema diff -ppassword

You will see an output similar to the following:

[INFO] Generating diff of 127.0.0.1:3306 development vs /home/database/production/*.sql
— instance: 127.0.0.1:3306
USE `development`;
\! /usr/bin/pt-online-schema-change –execute –alter ‘ADD COLUMN `deleteddate` datetime NOT NULL’ D=development,t=comments,h=127.0.0.1,P=3306,u=dbuser,p=XXXXX
[INFO] 127.0.0.1:3306 development: diff complete

This output shows that Skeema is ready to use pt-online-schema-change and you can continue to apply the change using the same command as above:

./skeema push -ppassword

Since this is an example and not a true production environment, then the change will apply quickly. If you used this process to change a very large and busy production environment database it could take a long time to alter a table.

While the change would take time, your application would remain online and not cause issues for your users, as the pt-online-schema-change tool will be copying data from the old to the new table behind the scenes.

Summary

In this post, we covered several build and deployment tools. We used approaches such as commands in our terminal to make changes as well as using third party services such as Travis to make repeatable build and deployment steps for us.

We looked at Packer, Ansible and Terraform working together to create a server, to copy and configure files and then replace an existing server with a new one. We also looked at altering database tables, a job that often comes hand in hand with developing web applications.

In our next post, we will look at backing up our databases which is a great process to have in place before altering large databases in production use.

An introduction to Infrastructure as Code with Terraform Cloudformation Ansible and Pulumi

In this post we will learn about Infrastructure as Code (IaC). IaC is the process of writing code that will take on the task of creating and maintaining your cloudbased infrastructure. This includes the items that make up your web application such as servers, security groups, network resources, and more. 

Some of the same tools and practices that you might use to develop and deploy application code changes into production, such as committing code changes to Github for testing and deployment, can also be used for Infrastructure. 

Using IaC tools, infrastructure requirements can be written in your code editor, just like normal application code. This work can be committed to a source code repository such as GitHub and undergo reviews from coworkers as well as run any tests before being deployed. 

A number of tools exist to help you to describe and develop the necessary infrastructure required for your software application. In this post, you will be introduced to working with a number of popular IaC tools, including: 

  • Hashicorp Terraform 
  • AWS CloudFormation 
  • Redhat Ansible  
  • Pulumi 

Hashicorp Terraform 

Terraform, created by a company called Hashicorp was released in 2014. It is a configuration language allowing you to describe the infrastructure items you would like to create using code. This code can then be stored in a repository to collaborate with a team and executed to create or modify the infrastructure. 

Terraform includes a concept known as Providers to allow Terraform to connect with cloud provider APIs such as AWS, Google, Azure, and many other cloud providers. 

Terraform is open source and uses a custom language called Hashicorp Configuration language (HCL) which is similar to JSON. 

Terraform code is Declarative, meaning you do not need to tell it the specifics of how to create an EC2 instance or a Security group, instead you tell it the EC2 instances or Security groups you want in place and it will create them as needed. If you run your Terraform code over and over again, it will create or modify your infrastructure as needed to get the desired result. It will not duplicate your EC2 instances or Security groups over and over again. 

If you modify your infrastructure independently of Terraform, such as resizing an EC2 instance manually in the AWS console, Terraform will be unaware of the change until you perform a terraform apply instruction. Terraform creates a ‘state’ file, which is a list of the items it manages so when you run Terraform again, it will compare its state file to their corresponding items in your account and it will plan out any additions, updates, or deletes that it needs to perform to bring the infrastructure back to the way you described it in your HCL code. 

If you begin to use Terraform, or any other Infrastructure as Code solution, it is best to commit to using it fully for the infrastructure you are creating. If you use both Terraform and also make changes manually in your cloud provider, Terraform will likely try to revert any changes you have made. In some cases this can be good, such as closing security group ports that are open however in some cases it might perform some actions you don’t like, such as resizing a server or database back down to a smaller size before you intentionally changed it manually. 

If you have an existing infrastructure that you have created before using Terraform, it is possible to import those items so that you can manage them with Terraform in the future. 

If you are importing an EC2 instance for example, you would need to write the HCL code first as if you are creating the instance for the first time, and then before running a terraform apply command, you would run a terraform import command. This would connect the EC2 instance you specify to the HCL code you have written and when it successfully imports, the information will be added to the state file and that EC2 instance can be modified using Terraform from that point onwards. 

When creating or importing any resources in Terraform such as an EC2 instance, you will need to know the required or optional arguments that Terraform will need so it can create a resource. The Terraform site provides detailed documentation for each AWS resource it can create on their site at https://www.terraform.io/docs/index.html 

Hashicorp, the creators of Terraform also provide a Learning resource available here https://learn.hashicorp.com/ and it includes a guide on how to install Terraform on Windows, Mac, or Linux based machines https://learn.hashicorp.com/terraform/getting-started/install.html. Follow the installation information and in the next section we will begin using Terraform to create a basic web server on AWS. 

We will use Terraform to create a key pair, a security group, an Ubuntu EC2 instance, and install Apache web browser and upload a simple Hello World HTML page. 

Using Terraform to create a simple web server on AWS 

To use Terraform we will create a small number of files. These file names don’t matter too much as long as they end with the .tf extension. These files will describe the items that we want to create in our AWS account and when we run Terraform, it will read each of the .tf files and evaluate which items need to be created and in what order to create them. 

We will use Terraform to create a simple web server, just like we used the AWS CLI in an earlier post. We will create an EC2 instance, a security group with one or more access rules, and a key pair to allow us to SSH into the server. 

We will create the following files: 

  1. main.tf – This is the main terraform file. It includes a Provider which tells Terraform we are working with AWS, the region to use, and the location of our credentials. 
  2. ec2.tf – this file will contain the instructions to create an EC2 instance using Ubuntu. 
  3. key_pair.tf – this will contain the instructions to create a Key Pair from your public key 
  4. security_group.tf – this will contain the instructions to create a security group and attach it to the EC2 instance 
  5. security_group_rules.tf – This will contain the instructions to open up port 80 so we can access the website on our EC2 server 
  6. variables.tf – This is where you define any variable types you’d like to use in your other terraform files 
  7. terraform.tfvars – this is where you store variable values that can be used in other terraform files 
  8. output.tf – this will contain the EC2 address and the security group ID that we have created so that we can connect to the server once it has been created. 
  9. files/index.html – a simple html page with the content ‘Hello World!’  

Next, we will look at each file in detail to show the content required for each file and describe its purpose when creating our web server on AWS 

main.tf 

This is the main terraform file. It includes a Provider which tells Terraform we are working with AWS, the region to use, and the location of our credentials. 

provider “aws” {
  region             
= “us-east-1”
 
shared_credentials_file = “~/.aws/credentials”
  profile            
= “example”
}
 

Ec2.tf 

This file contains the instructions to create an EC2 instance using Ubuntu. The Data block performs a search for an Ubuntu AMI on the AWS marketplace that can be used by the aws_instance block later in the file. 

data “aws_ami” “ubuntu” {
 
most_recent = true  filter {
name   = “name”

values = [“ubuntu/images/
hvm-ssd/ubuntu-bionic-18.04-amd64-server-*”]
 
}  filter {
name   = “virtualization-type”

values = [“
hvm“]
  }

  owners = [“099720109477”] # Canonical

}
 

 # Create EC2 instance
resource “
aws_instance” “example” {
 
ami                = “${data.aws_ami.ubuntu.id}”
 
instance_type      = “t3.micro
 
vpc_security_group_ids = [“${aws_security_group.example.id}”]
 
key_name           = “${aws_key_pair.pem-key.key_name}”

  tags = {
Name = “terraform-webserver”

  }

}
 

key_pair.tf 

This file contains the instructions to create a Key Pair from your public key. It assumes you have a public key file on your PC located at ~/home/.ssh/id_rsa.pub. 

resourceaws_key_pair” “pem-key” {
 
key_name   = “terraform-webserver-key”
 
public_key = “${file(“~/.ssh/id_rsa.pub”)}”
}
 

security_group.tf 

This file contains the instructions to create a security group and attach it to the EC2 instance. We do not open ports to allow traffic in to the EC2 instance until the next file. 

resource “aws_security_group” “example” {
  name   
= “example”
  description = “example security group”

  tags = {
Name = “terraform-webserver”
  }

}
 

security_group_rules.tf 

This file contains the instructions to open port 80, so we can access the website on our EC2 server from our web browser. This file also contains a rule to open port 22 so that we can SSH into the server if needed. 

Finally, an egress rule is added to allow all traffic out of the EC2 instance. 

resource “aws_security_group_rule” “http” {
  type         
= “ingress”
 
from_port     = 80
 
to_port       = 80
  protocol     
= “tcp
 
cidr_blocks   = [“0.0.0.0/0”]
 
security_group_id = “${aws_security_group.example.id}”
  description  
= “Public HTTP”
}
 

resource “aws_security_group_rule” “ssh” {
  type         
= “ingress”
 
from_port     = 22
 
to_port       = 22
  protocol     
= “tcp
 
cidr_blocks   = [“${var.my_ip_address}/32″]
 
security_group_id = “${aws_security_group.example.id}”
  description  
= “SSH access”
}
 

resource “aws_security_group_rule” “example_egress” {
  type         
= “egress”
 
from_port     = 0
 
to_port       = 0
  protocol     
= “all”
 
cidr_blocks   = [“0.0.0.0/0”]
 
security_group_id = “${aws_security_group.example.id}”
  description  
= “Allow all out”
}
 

variables.tf 

This is where you define any variable types you’d like to use in your other terraform files. They are like variables exported from a function in a program to be used elsewhere in your  code 

# Variables used
variable “
my_ip_address” {
  // read from
terraform.tfvars
  type = “string”
} 

terraform.tfvars 

This file is where you create any variable values that can be used in other terraform files. For now, we just need to include one variable called my_ip_address, which should contain your own IP address so that your IP address will be allowed to SSH in to the resulting EC2 instance. 

# Your current IP address
my_ip_address = “xxx.xxx.xxx.xxx 

output.tf 

This will contain the EC2 address and the security group ID that we have created so that we can connect to the server once it has been created. 

output “public_dns” {
  description = “The EC2 instance DNS”

  value  
= “${aws_instance.example.public_dns}”
}
 

output “security_group_id” {
  description = “The security group ID”

  value  
= “${aws_security_group.example.id}”
}
 

Files/index.html 

Hello World! 

Here we will create a simple html file with the content of  ‘Hello World!’ or whatever content you would like to see when we have created our first web server  and access it via our web browser to confirm it is responding with our chosen content. 

Running our Terraform files  

Now that we have created out files and we have learned a little bit about the purpose of each file, it is time to use Terraform review the plan of the items it will create and then process to create our infrastrucutre. 

Before we being running Terraform, may sure you have edited the terraform.tfvars file to include your current IP address. This will allow you to SSH into the server later and to run a Provisioner to customize the server. 

Once the files are in place, run the following command to initialize Terraform. 

terraform init 

This step will download any necessary providers, in this case it is an AWS provider so that Terraform can communicate with the AWS API. 

You can use the following command to view the steps that Terraform plans to take, without making any actual changes: 

terraform plan 

If there are any issues with your *.tf files, you will begin to see those errors here in the output. The error output should show you the file with the problem and the line number to help you to debug any issues. 

If there are no issues, the output will show you the changes it plans to make. The changes are broken down into three categories, those items to add, change, or destroy. Since this is our first run, it should only have items to add. 

Plan: 5 to add, 0 to change, 0 to destroy. 

If you are satisfied with the output, we can use the following command to tell Terraform to go ahead and apply the changes to your AWS account 

terraform apply 

When you run this command, the changes will not be made straight away. Terraform will review its State file and compare it to the current changes to make and it will show the changes it plans to make again, just like the terraform plan command. 

This time it will prompt you to ask you if you’d like to go ahead. Type yes and press enter to continue to let Terraform make the changes. Since this is a small example, it should only take a minute or less to apply. 

Once it has completed, it will show you a small output based on the instructions we added to the output.tf file. It will show the public DNS name of the new server that has been created. 

Congratulations, you just used Terraform to create a server on AWS. If you log in to your AWS console in your web browser, you’ll see a new EC2 instance called terraform-webserver running in the us-east-1 region. 

So that Terraform can remember the items it has created, it creates a file called terraform.tfstate. You can open the file and take a look. It stores information on the items it has created so far in a JSON format. Don’t delete or alter this file. If you delete the file, then Terraform will no longer remember the items it has created in your AWS account and if you run terraform apply, it will try to create all of the items again. 

If you try running terraform plan or terraform apply once again, you’ll see that Terraform doesn’t want to make any changes. This is because Terraform has made the necessary changes and the items now in your AWS account match the information stored in the local Terraform state file. 

In its current state, this new server isn’t of much use. It is online and working, however it isn’t doing anything. Let’s go ahead and use Terraform to install Apache web server and display a ‘Hello World’ web page on the server. 

Using Terraform to configure our webserver 

To do this, we will use a Provisioner. This will allow us to run one or more commands on the EC2 instances on its first run only. 

In the ec2.tf file, add the following content within the aws_instance resource, just after the tags block:  

resource “null_resource” “webserver” {
  provisioner “file” {

// upload the index.html file

source = “files/index.html”

destination = “/home/ubuntu/index.html”

  }
 

provisioner “remote-exec” {
inline = [

 
sudo apt update”,
 
sudo apt install apache2 -y”,
 
sudo mv /home/ubuntu/index.html /var/www/html/index.html”
]

  }
 

connection {
// connect over
ssh
type   
= “ssh
user   
= “ubuntu”
private_key
= “${file(“~/.ssh/id_rsa“)}”
timeout
= “1m”
port   
= “22”
host   
= “${aws_instance.example.public_dns}”
  }

}
 

This file will use a null_resource to upload our index.html file and then use a remote-exec provisioner to run an apt update, install Apache, and move the index.html file into place. 

The connection block tells Terraform to connect to the server using SSH and provides the username, private key, port, and the host to connect to. 

If we perform a terraform apply command now, nothing will happen as our EC2 instance has already been created from our earlier step. 

We can instruct Terraform to replace the existing EC2 instance and create a new one by tainting the instance. This will mark the instance for replacement the next time Terraform is applied. 

Use the following command to taint our existing EC2 instance: 

terraform taint aws_instance.example  

It should show an output as follows:  

Resource instance aws_instance.example has been marked as tainted. 

Now, we can perform an apply to recreate this instance and it will run the above provisioner as part of its creation. 

terraform apply 

Once this has been completed, open up the Public DNS in your browser and you should see our ‘Hello World!’ HTML file.

In this example, we have used a Provisioner to customize our Web server, to install Apache and upload our HTML file. In a Production environment, using a Provisioner is not recommended as it can add uncertainty and complexity to the process. Hashicorp recommends running configuration management software instead, such as Packer. Packer allows you to create a server image first, customize and save it as an AMI, and then Terraform can take this saved image and use it to create an already configured EC2 instance. We will cover using Packer later in this book. 

The files for this webserver created using Terraform are available here: https://github.com/gordonmurray/terraform_webserver 

If you would like to remove the items that Terraform has created in your AWS account, you can use the following command: 

terraform destroy 

You will be prompted again to add yes and press enter to continue. Terraform will then go ahead and remove any items it created. Once it has finished, you will see the output similar to: 

Destroy complete! Resources: 5 destroyed. 

This webserver example is a basic working example, using the Hashicorp documentation available at https://www.terraform.io/docs/index.html you can improve on the existing files to customize them to your needs. 

As a suggested next step, try to create a static IP address and attach it to your EC2 instance, so that the IP address of the server don’t change any time you change or replace the server. Use the following documentation page as a guide to get started https://www.terraform.io/docs/providers/aws/r/eip.html 

AWS CloudFormation 

CloudFormation was launched by AWS in 2011 It is a service which is specific to AWS. CloudFormation can only be used to create AWS resources, unlike Terraform which can use Providers to work with many different cloud providers. 

Using CloudFormation itself is free, though the resources it creates on AWS have their respective costs. CloudFormation has a useful feature built in to estimate your AWS costs. When you are preparing to create the ‘stack’ it can estimate the cost by linking to the AWS calculator and pre-populating it the items from your CloudFormation stack, so you can see the estimated cost before you proceed. 

Using AWS CloudFormation to create a simple webserver on AWS 

CloudFormation gives you a number of options to begin to create your Stack. You can upload or import a template from S3, use a premade template, or create one using the Designer interface in the browser. CloudFormation supports using either JSON or YAML and in our examples we will use YAML. 

YAML stands for YAML Ain’t Markup Language. We will use it in this book as it is more human readable when compared to JSON. YAML is also a common configuration language used by multiple tools, such as Ansible or Kubernetes, so we will see more YAML as we progress through this book. Comments can also be added to YAML files, unlike JSON files. 

CloudFormation cannot create an AWS Key Pair directly. Instead it asks to use an existing Key Pair. If you don’t already have a Key Pair in place in your AWS account, you can log in to the AWS console to create one, or use the following CLI command to create one in the us-east-1 region:  

aws ec2 create-key-pair –key-name example –query ‘KeyMaterial‘ –output text example.pem  —region us-east-1 

Next, we will begin creating the files that will contain the instructions we want to give to CloudFormation. First, we will initialize our template and set some parameters that will be needed by CloudFormation to create our stack. 

The AWSTemplateFormatVersion field at the beginning of the file is optional, it identifies the version of the CloudFormation template so it can be interpreted correctly by CloudFormation. 

We use the Parameters to tell CloudFormation that we will need a KeyName, a VPCID, and a SubnetID. CloudFormation will ask you to fill in those three values before it begins to create the stack. 

Create a file called webserver.yaml with the following content: 

AWSTemplateFormatVersion: “2010-09-09”
Description: Creates an EC2 instance and a security group

Parameters:

KeyName
:
   
Description: “Name of an existing EC2 KeyPair to enable SSH access to the instance”
   
Type: “AWS::EC2::KeyPair::KeyName
   
ConstraintDescription: “must be the name of an existing EC2 KeyPair
VPCID:

   
Description: “VPC IDs”
   
Type: “AWS::EC2::VPC::Id”
SubnetIDs
:
   
Description: “Subnet IDs”
   
Type: “AWS::EC2::Subnet::Id” 

Next, we will use a required Resources section to create an AWS Security group. This security group will open Port 22 for SSH access and HTTP port 80 for web traffic. 

The optional Tags section allows you to add one or more tags to identify your security group. 

Resources:
InstanceSecurityGroup
:
   
Type: “AWS::EC2::SecurityGroup
   
Properties:
       
Tags:
           
– Key: “Name”
             
Value: “cloudformation-webserver”
       
VpcId: !Ref VPCIPsVPCID
       
GroupDescription: “Webserver security group”
       
SecurityGroupIngress:
           
CidrIp: “0.0.0.0/0”
             
Description: “Allowed from anywhere”
             
FromPort: “22”
             
ToPort: “22”
             
IpProtocol: “tcp
           
CidrIp: “0.0.0.0/0”
             
Description: “HTTP access from anywhere”
             
FromPort: “80”
             
ToPort: “80”
             
IpProtocol: “tcp 

Here, we will append to the existing Resources section to create our EC2 webserver instance, below our existing Security group text. It will use an AMI called ami-02df9ea15c1778c9c, which is a freely available Ubuntu instance. 

This EC2 instance will use the security group we created already earlier in our CloudFormation stack.  

The SecurityGroupIds section is being told to get the Resource called InstanceSecurityGroup and its value of GroupId. 

Again, we have included optional Tags section to identify out EC2 instance 

Finally, we use the UserData section to embed a simple bash script to run on our EC2 instance when it is created for the first time. 

In this example, the user data will perform an apt update to update the Ubuntu instance, it will install Apache Web server and place a HTML file with the content of Hello World! in the /var/www/html folder for Apache to show. 

EC2Instance:
   
Type: “AWS::EC2::Instance”
   
Properties:
       
ImageId: “ami-02df9ea15c1778c9c”
       
InstanceType: “t2.micro
       
SecurityGroupIds:
           
– !GetAtt InstanceSecurityGroup.GroupId
       
KeyName: !Ref KeyName
       
SubnetId: !Ref SubnetIDs
       
Tags:
           
– Key: “Name”
             
Value: “cloudformation-webserver”
       
UserData:
           
Fn::Base64: !Sub |
               
#!/bin/bash –xe
               
sudo apt update
               
sudo apt install apache2 -y
               
echo “Hello World!” > /var/www/html/index.html 

AWS provide a way to validate your CloudFormation YAML files. Using the AWS CLI, which we used earlier in the book, you can run the following command to validate your webserver.yaml file: 

aws cloudformation validate-template –template-body file:///home/webserver.yaml 

Now that we have written the necessary file and used the command line to validate it, we are now ready to use the AWS console to create our CloudFormation Stack. 

If you are not logged in to your AWS console, log in now and we will create our CloudFormation Stack in the next section 

Running CloudFormation using the AWS console 

Log in to the AWS console, search for CloudFormation, open it and click on Create Stack. 

You will progress through several screens as follows: 

  1. Select the Template is ready option from the Prerequisite section 
  2. Select the Upload a template file from the Specify template section and then Click on Choose file and select your webserver.yaml from above
  3. Upload your webserver.yaml file and click on Next. 
  4. Enter a Stack name, such as Webserver. 
  5. Choose a KeyName from the dropdown menu. 
  6. Choose a SubnetID from the dropdown menu. 
  7. Choose a VPCID from the dropdown menu and click on Next.
  8. Add some optional Tags and choose an IAM role with permissions to create EC2 instances. Leave the IAM role as cloudformation so that CloudFormation has permission to create the resources then click Next.
  9. Use this Review Webserver page to ensure your choices from previous steps are OK, and then click on Create Stack at the bottom of the page.
  10. Your stack should now show a Status of CREATE_IN_PROGRESS
  11. If the initial deployment of the stack fails, you will need to delete the Stack and start again. Once the Stack has built successfully, you can perform any updates without deleting and restarting.
  12. Once the Stack has deployed fully, you can click on the Resources tab to see the items it created. 

Congratulations, you have now used CloudFormation to create a simple web server! 

The source code for the above is available here : https://github.com/gordonmurray/cloudformation_webserver. 

If you wish to delete your Stack afterwards, you can select the radio button next to our webserver stack name and then click Delete. It will remove the items created as part of the Stack deployment.

When deleting a Stack, CloudFormation will delete every Resource described in your Stack. This includes the EC2 instance and its Security Group. 

Running CloudFormation using the AWS CLI 

Before using the AWS CLI to create the Stack, you will need to know the values of each of the following that you would like to use.

  1. VPC ID 
  2. Subnet ID 
  3. Key Pair name 

The following command will use the AWS CLI to create a CloudFormation stack along with parameters to include our template file, KeyName value, SubnetID value, and VPCID value in the AWS region of us-east-1. 

aws cloudformation create-stack –stack-name webserver-cli –template-body file://webserver.yaml –parameters ParameterKey=KeyName,ParameterValue=[key pair name here] ParameterKey=SubnetID,ParameterValue=[ subnet value here ] ParameterKey=VPCID,ParameterValue=[ VPC ID value here] –region us-east-1 

If the command has executed properly, you will receive an output like the following: 

{
StackId“: “arn:aws:cloudformation:us-east-1:123456789:stack/webserver-cli/12345-1111-2222-3333-1234567890″
} 

If you would like to delete the stack afterwards, you can use the following 

aws cloudformation delete-stack –stack-name webserver-cli –region us-east-1 

In this section, we have created a webserver stack using AWS CloudFormation. We prepared the webserver YAML file so that CloudFormation knew the resources it needed to create. 

We used both the AWS console to create the stack step by step as well as gained experience using the AWS CLI to create a stack also. 

In our next section, we will use Ansible to create and configure a server on AWS. You will see that Ansible also uses YAML to create and configure resources. 

Ansible 

Ansible was originally released in 2012 and in 2015 was acquired by Red Hat. Ansible is primarily a configuration management tool that can be used to configure both Windows and Linux environments. 

Using additional modules, Ansible can also manage infrastructure on cloud providers such as AWS. It can be a good choice for creating both infrastructure and then also applying any configuration to servers you create within the infrastructure, allowing you to use the one tool to do both tasks. 

Ansible also includes Ansible Vault. Vault allows you to encrypt files containing sensitive content such as passwords or API keys, so that they can be committed to a source code repository without being stored in plain text. Ansible uses Advanced Encryption Standard (AES) to encrypt the data and you will need a password to edit the data. 

You can store your password to decrypt the files as an environment variable so that they can be read by Ansible during a build or deployment process. We will cover build/deployment processes in our next post. 

If you already use Ansible to configure items within your infrastructure, it can be an attractive choice to also use Ansible to build infrastructure. Since you already know the language, you don’t need to learn anything new, however there are a couple of items to know in advance. 

When creating infrastructure, Ansible doesn’t record any State. This can sometimes lead to difficulties when changing your infrastructure as it doesn’t know of a previous state to compare to. If you terminate an EC2 instance for example and run your Playbook to re-create a new EC2 instance of the same name, it will fail to build as the previous server name still exists for a short time until it has fully terminated. It is possible to work around it with unique server names, though you will need to take additional steps to do so. 

Finally, if you want to remove some infrastructure, Ansible doesn’t have a built-in process to taint or remove existing resources such as EC2 instances. You will need to write additional code to remove those items specifically. 

Using Ansible to create and configure a webserver on AWS 

To install Ansible, use the Ansible documentation here to install a version of Ansible for your system: https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#installing-the-control-node. 

The following examples use Ansible version 2.8.3 and Python version 3.7.5 and will create an Ubuntu 18.04 LTS server in the US region.  

If you would like to change to another Ubuntu distribution, you can find its AMI here: https://cloud-images.ubuntu.com/locator/ec2/. 

The following steps assume there is an existing VPC and at least one subnet which are available by default in a new AWS account. 

To use the EC2 module, we will also need to install a small number of dependencies. We need to install Python as Ansible is written using Python and needs Python installed locally to run 

sudo apt install python 

We install python-pip, which is a package manager for Python to install additional modules

sudo apt install python-pip

We install boto, a python SDK to help Python to interact with AWS resources. 

pip install boto boto3 

Now that we have installed the necessary dependencies, let’s go ahead and  create two main Ansible playbooks. The first will create a security group and an EC2 instance and add tags to both. The tags are important for the next step of configuring the server. The second playbook will connect to the EC2 instance and install Apache and upload a simple index.html file, just like our other examples in this book such as creating a webserver using AWS CloudFormation. 

We will create the files and folders shown in the following screenshot. These files and folders follow the recommended structure of an Ansible Role. Organizing playbooks with Roles allow you to reuse code in other Ansible Playbooks in the future.  

Before we begin using Ansible to create resources in AWS, we will need an AWS key and secret so that Ansible has permission to create resources. 

If you have the AWS CLI already installed, you can either use your existing AWS Key and secret usually stored in ~/.aws/credentials, or create a new API key for this example. 

To use the AWS CLI to create a new access key and secret, use the following command: 

aws iam create-access-key 

You will see a JSON output, take note of the AccessKeyId and SecretAccessKey values. 

In your command line, type the following to get Ansible Vault to create a secure file: 

ansible-vault create group_vars/all/pass.yml 

You will be asked to supply a password and to confirm the password to make sure it is correct. 

This command will create a new empty file and the file will be opened up to begin adding content. Once the file is open, add the following: 

group_vars/all/pass.yml 

ec2_access_key: [ place your AccessKeyId here ]
ec2_secret_key: [ place your
SecretAccessKey here ] 

If you’d like to edit this file at any time, use the following Ansible-vault command: 

ansible-vault edit group_vars/all/pass.yml 

Next, lets look at each of the files shown in the previous image that Ansible will need to create our resources 

Infrastructure.yml 

This Infrastructure playbook file creates an EC2 instance and a Security Group. Items are Tagged so that our next Playbook can find and configure them 


– hosts: localhost

  connection: local

 
gather_facts: false
vars:

    region: eu-west-1

    image: ami-04c58523038d79132 # Ubuntu 18.04 LTS

roles:

    – keypair

    – ec2

    –
securitygroup

Webserver.yml 

Our webserver playbook configures an EC2 instance to become a webserver. The EC2 instance was Tagged so that this Playbook can configure the server 


– hosts: all

 
gather_facts: no
 
become: yes
  user: ubuntu

  roles:

    – apache

    – php
 

Src/index.php 

A simple file to show the webserver is operating and showing files. 

<h1>Hello World from Ansible!</h1> 

aws_ec2.yml 

This is an Ansible plugin. It is a Dynamic inventory source. It can query your EC2 account for Tagged instances, instead of hard coding instance IP addresses. 

plugin: aws_ec2
regions:

  – eu-west-1

filters:

 
tag:Name: Webserver
aws_access_key_id
: [ your aws access key ]
aws_secret_access_key
: [ your aws secret key ] 

In the highlighted code above, make sure to include your own AWS Key and Secret values  

group_vars/all/pass.yml 

This file holds AWS credentials so Ansible can connect to your AWS account 

ec2_access_key: [ your aws access key ]
ec2_secret_key: [ your
aws secret key ] 

Roles/apache/files/webserver.conf 

This is an Apache virtual host config file. It tells Apache where to look for your webserver files. 

<VirtualHost *:80>
    
ServerAdmin webmaster@localhost
   
DocumentRoot /var/www/html
    
ErrorLog ${APACHE_LOG_DIR}/error.log
   
CustomLog ${APACHE_LOG_DIR}/access.log combined
     <Directory /var/www/html>

        Options Indexes
FollowSymLinks
       
AllowOverride All
        Require all granted

    </Directory>

</
VirtualHost> 

Roles/apache/tasks/main.yml 

This is an Ansible task within a Role, to install Apache. APT is used to install Apache and some common steps are added to remove the default Apache homepage and disable the default Apache virtual host. We then add our own Apache virtual host and restart Apache for the changes to take effect. 

– name: Install Apache
  vars:

    packages:

      – apache2

  apt:

    pkg: “
{{ packages }}”
   
update_cache: yes
    state: latest

– name: Delete the default apache landing page
  file:

    path: /var/www/html/index.html

    state: absent

– name: Disable apache2 default vhost
  file:

    path: /
etc/apache2/sites-enabled/000-default.conf
    state: absent
 

– name: Copy our virtual host
  copy:

   
src: webserver.conf
   
dest: /etc/apache2/sites-available/webserver.conf
  register:
host_uploaded 

– name: Enable our virtual host
  shell: /
usr/sbin/a2ensite webserver.conf
  when:
host_uploaded.changed 

– name: Restart Apache
  service:

    name: apache2

    state: restarted

  when:
host_uploaded.changed 

Roles/deploy/tasks/main.yml 

This is an Ansible task within a Role, to upload our /src folder to the webserver. Any files that we add to the /src folder, such as the pages necessary for a website, will be uploaded to the /var/www/html folder for Apache to serve.  

– name: Copy our src directly to /var/www/html
  copy:
    src: ./src/
   
dest: /var/www/html/ 

Roles/ec2/tasks/main.yml 

This is an Ansible task within a Role, to create an EC2 instance and tag it. We provide a key pair name, the instance type, and security group ID to use when creating the EC2 instance. We also tag the instance so that Ansible can find the instance later to configure it. 

– name: Provision an EC2 instance
  ec2:

   
aws_access_key: “{{ ec2_access_key }}”
   
aws_secret_key: “{{ ec2_secret_key }}”
   
key_name: “ansible-webserver”
    id: “ansible-web-server”

   
group_id: “{{ security_group.group_id }}”
    image: “
{{ image }}”
   
instance_type: t2.micro
    region: “
{{ region }}”
   
wait: true
    count: 1

  register: webserver
 

– name: Tag the webserver EC2 instance
  ec2_tag:

   
aws_access_key: “{{ ec2_access_key }}”
   
aws_secret_key: “{{ ec2_secret_key }}”
    region: “
{{ region }}”
    resource: “
{{ webserver.instances[0].id }}”
    state: present

    tags:

      Name: Webserver
 

Roles/keypair/tasks/main.yml 

This is an Ansible task within a Role, to create and AWS Key Pair. We give the key pair a name and tell it to read our public key located at ~/.ssh/id_rsa.pub so that we can successfully connect to the webserver using SSH later.  

– name: Upload public key to AWS
  ec2_key:

    name: “ansible-webserver”

   
key_material: “{{ lookup(‘file’, ‘~/.ssh/id_rsa.pub’) }}”
    region: “
{{ region }}”
   
aws_access_key: “{{ ec2_access_key }}”
   
aws_secret_key: “{{ ec2_secret_key }}” 

Roles/php/tasks/main.yml 

This is an Ansible task within a Role, to install PHP. Additional packages could be added to the list here if needed, such as php-curl. This tells the server to install each package using APT. 

– name: Install PHP and some related packages
  vars:

    packages:

      – php

  apt:

    pkg: “
{{ packages }}”
   
update_cache: yes
    state: latest
 

Roles/securitygroup/tasks/main.yml 

This is an Ansible task within a Role, to create a security group and tag it. We give the security group a name, a description, and open up ports 22 for SSH access to the server and port 80 for HTTP access to the webserver. We also open an egress (outbound) rule to allow traffic out of the instance. 

– name: Create a security group
  ec2_group:

    name: “webserver-security-group”

    description: “Security group for Webserver”

    region: “
{{ region }}”
   
aws_access_key: “{{ ec2_access_key }}”
   
aws_secret_key: “{{ ec2_secret_key }}”
    rules:

      – proto:
tcp
        ports:

          – 22

       
cidr_ip: 0.0.0.0/0
       
rule_desc: allow all on ssh port
      – proto:
tcp
        ports:

          – 80

       
cidr_ip: 0.0.0.0/0
       
rule_desc: allow all on HTTP port
   
rules_egress:
      – proto: “all”

       
cidr_ip: “0.0.0.0/0”
  register:
security_group

– name: Tag the Security group
  ec2_tag:

   
aws_access_key: “{{ ec2_access_key }}”
   
aws_secret_key: “{{ ec2_secret_key }}”
    region: “
{{ region }}”
    resource: “
{{ security_group.group_id }}”
    state: present

    tags:

      Name: Webserver  
 

The above files can be found at: https://github.com/gordonmurray/ansible_webserver_aws 

To create the AWS infrastructure items, use the following command in your terminal in the same folder as the Ansible files we have created earlier. 

ansible-playbook infrastructure.yml –ask-vault-pass 

To configure the webserver once it has been created, use the following command:  

ansible-playbook –i aws_ec2.yml webserver.yml –ask-vault-pass 

You will be asked to supply your password each time so that Ansible can decrypt the pass.yml file. 

In the Ansible output, you will see the public DNS address of the new EC2 server, it will be in the format of ec2-111-111-11.11.eu-west-1.compute.amazonaws.com. 

Open this address in your browser, and you should see Hello World! 

Congratulations, you just used Ansible to create and configure a server on AWS. 

In our next section, we will look at Pulumi. When compared to CloudFormation, Terraform, or Ansible, Pulumi is relatively new to the industry of infrastructure as code. Its approach is different to the declarative nature of the tools we have used so far which makes it especially interesting to look at. 

Pulumi 

Pulumi is very new compared to the tools we have used so far. It was launched in June of 2018. It is free to use via their community edition. Pulumi’s code is open source and available on their GitHub site at https://github.com/pulumi/pulumi. 

In the same way that Terraform has providers to connect to other online cloud services, Pulumi also includes providers, with a growing list of services listed here on their site https://www.pulumi.com/docs/intro/cloud-providers/. 

Pulumi also includes a resource called Cross Guard. Cross Guard is a way to enforce business rules or policy through code. A business might allow their developers to create and manage their infrastructure but may wish for certain rules to apply to the items the developers are creating. 

As an example, a business may like to apply limits such as allowed EC2 instance sizes to manage costs, or ensure some security policies are in place at all times in the infrastructure.  

There is great potential in Pulumi, it can help bring developers, operations, and even compliance teams closer together by sharing a codebase in a true DevOps fashion.  

In our next section, we will use Pulumi with Python to create a Webserver on AWS. 

Using Pulumi to create a webserver on AWS 

One of the things that makes Pulumi unique when compared to CloudFormation, Terraform, or Ansible is that with Pulumi you can use a programming language you may already be familiar with to create and manage your infrastructure.  

At the time of writing this, Pulumi supports the following runtime languages:

  • Node. js – JavaScript, TypeScript, or any other Node. js compatible language. 
  • Python – Python 3.6 or greater. 
  • . NET Core – C#, F#, and Visual Basic on . NET Core 3.1 or greater. 
  • Go – statically compiled Go binaries. 

Pulumi also works with several core cloud providers including AWS, Google Cloud, and Azure.  

Creating a Pulumi Stack using Python 

Earlier in this post, we saw that AWS CloudFormation uses the term stack to describe the infrastructure items that CloudFormation will create. Pulumi also uses this same term to describe an infrastructure project. 

The first step is to download and install Pulumi. There is a great Getting Started guide on their website at https://www.pulumi.com/docs/get-started/, which will guide you on how to install Pulumi for your platform and choose the language runtime. 

In the steps that follow, we will use AWS, running from a Linux platform using Python. 

If you have the AWS CLI installed from previous sections of this book, you’re good to go. If you don’t have the AWS CLI installed, then take a moment to go to the AWS website and follow the steps to install the CLI https://aws.amazon.com/cli/. 

We need to make sure our system has the necessary Python version and packages to continue, so we will run the following command in our Terminal first: 

sudo apt install python-dev -y
sudo
apt install python3-pip -y
sudo
apt install python3-venv -y 

python-dev contains everything Python needs to compile python extension modules. python3pip makes sure we have the PIP, a Python Package manager installed for Python version 3, and python3-venv provides support for creating lightweight virtual environments. Each virtual environment has its own Python binary and can have its own independent set of installed Python packages in its site directories. 

Once we have installed the necessary Python packages, we can create our first Pulumi Stack: 

First, we will make a folder to store our project: 

mkdir webserver-pulumi && cd webserver-pulumi 

Next, we will run Pulumi with a number of parameters to set up our project: 

pulumi new aws-python –name webserver-pulumi –stack webserver 

This tells Pulumi to create a new project structure, using AWS and Python. We use the –name parameter to give our project a name of webserver-pulumi and we use –stack to give the stack a name of webserver. 

When creating a new project, Pulumi will ask you a short number of questions in your terminal 

  • Project description – This allows you to give your project a short description. You can leave the default text in place or add something new like A sample Pulumi project. 
  • AWS region – This allows you to choose an AWS region to use to deploy your stack. You can leave the default us-east-1 in place or change it to a new region such as eu-west-1 if you prefer. 

Pulumi will create a small number of files in our webserver-pulumi directory: 

  • __main__.py – this is the main file that we will use and add more Python code to shortly to describe our stack 
  • pulumi.webserver.yaml – This file holds any config items for Pulumi. It will show the aws region choice we made in our earlier step. You can change the region if you like but there is no need to. 
  • pulumi.yaml – This file shows the meta data of our project including its name, runtime, and description. You can change the region if you like but again there is no need to. 
  • requirements.txt this file contains a short list of items Python will need to import to use to create our stack with Pulumi. 

Once our stack has been created with the above files, Pulumi will give us three steps to follow to create a virtual environment and install the requirements 

python3 -m venv venv
source
venv/bin/activate
pip3 install -r requirements.txt
 

Run each of the commands in the order they appear.  

You are now ready to run Pulumi by running the pulumi up command. By default, the __main__.py file will have some existing code to try to create an S3 bucket on AWS. 

If you run Pulumi, it will show you a list of the items it wants to create. You will be given a list of options including yes, no, and details. 

  • yes will go ahead and create the S3 bucket on AWS 
  • no will simply end the command and return you to your command prompt 
  • details will show you some more details on the items that will be created 

If you use yes, Pulumi will create the S3 bucket. Since s3 bucket names need to be unique, it will automatically append a string to the my-bucket name if you left the default value in place. 

To continue on with creating a webserver, lets remove the s3 bucket and continue with the next steps. Use the following command to delete your existing stack: 

pulumi destory 

Pulumi will show you a list of the items it will create, you can use yes, no, or details again. Choose yes to allow Pulumi to go ahead and delete the stack resources. 

At this point, the stack still exists though the s3 bucket has been removed. In our next step, we can add some additional information to the __main__.py file to create our webserver. 

Open __main__.py in an editor an perform the following steps:

Empty the file of any content, leaving only the first two lines of : 

import Pulumi
from
pulumi_aws import s3

We will add some variables that Pulumi will need to create our stack. We will add size, vpc_id, subnet_id, public_key, and user_data, very similar to our earlier examples when using the AWS CLI and CloudFormation, so that Pulumi knows the values to use. We will add some minimal user data that the EC2 instance will run when it is created. 

# Define some variables to use in the stack
size = “t2.micro”
vpc_id = “vpc-xxxxxx
subnet_id = “subnet-xxxxxx
public_key = ” [ Add your public key content here, usually found in ~/.ssh/id_rsa.pub”
user_data = “””#!/bin/bash
sudo apt update
sudo apt install apache2 -y
echo ‘Hello World!’ /var/www/html/index.html”””

Next, we will add a get_ami block to tell Pulumi to search for an Ubuntu AMI on AWS to use when creating our EC2 instance.  

# Get the AMI ID of the latest Ubuntu 18.04 version from Canonical
ami = aws.get_ami(
most_recent=”true”,
owners=[“099720109477”],
filters=[{
“name”: “name”,
“values”: [“ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-*”],
}],)

We will use KeyPair to create a new Key Pair on AWS so that we can connect to the EC2 instance via SSH later if we want to. We provide a key pair name of keypair-pulumi and read in the variable called public_key which we defined earlier to attach our public key value. 

# Create an AWS Key Pair
keypair = aws.ec2.KeyPair(
“keypair-pulumi“,
key_name=”keypair-pulumi“,
public_key=public_key,
tags={“Name”: “keypair-pulumi“})

Next, we will create a Security group and add three rules. We will open port 22 so that we can SSH into the server. In the ingress block, we will open port 80 so that the server can be accessed over HTTP in a web browser. In the egress block, we will add a single entry for -1, which means all traffic, to allow the EC2 instance to send outbound traffic. We will give the security group a name of securitygroup-pulumi and provide a description also. It also used the vpc_id variable we declared earlier to create the security group in the correct VPC. 

# Create an AWS Security group with ingress and egress rules
group = aws.ec2.SecurityGroup(
securitygroup-pulumi“,
description=”Enable access”,
vpc_id=vpc_id,
ingress=[{
“protocol”: “tcp“,
from_port“: 22,
to_port“: 22,
cidr_blocks“: [“0.0.0.0/0”],},
{“protocol”: “tcp“,
from_port“: 80,
to_port“: 80,
cidr_blocks“: [“0.0.0.0/0”],},
],
egress=[{“protocol”: “-1”, “from_port“: 0, “to_port“: 0, “cidr_blocks“: [“0.0.0.0/0”],}],
tags
={“Name”: “securitygroup-pulumi“},)

Finally, we add the interesting part, we will use ec2.instance to create the actual EC2 instance. We give the server a name of webserver-pulumi, we use the size parameter to control the size of the instance. In vpc_security_group_ids, we reference the security group created in the previous step by its block name of group and its value id so the EC2 instance knows the Security group ID to use. 

We pass in user_data so that the EC2 instance will run a bash script when it starts up for the first time. This bash script installs Apache webserver and creates a simple hello World file. 

The ami variable is the Ubuntu AMI we declared at the top of the file. The key_name references the key file we created in a previous step, just like the security group.  

We use the subnet_ids variable to apply the subnet variable we declared at the top of the file also. Finally, we apply some optional tags so we can name the server. 

# Create the webserver EC2 instance
server = aws.ec2.Instance(
“webserver-pulumi“,
instance_type=size,
vpc_security_group_ids=[group.id],
user_data=user_data,
ami=ami.id,
key_name=keypair.key_name,
subnet_id=subnet_id,
tags={“Name”: “webserver-pulumi“},)

Our last step for this file is to add a simple export. This will export the Public IP address of our EC2 instance so that when Pulumi creates our Stack, it will write the EC2 instance Public IP address to our screen, so we can see it and copy it in to our web browser to try it. 

# Show the Public IP address of the webserver EC2 instance
pulumi.export(“publicIp“, server.public_ip) 

With the __main__.py file ready, we can now run the following to get Pulumi to create the Key Pair, the security group, and the EC2 instance 

pulumi up 

Like earlier, Pulumi will show us the items it is going to create. Anwser yes to allow Pulumi to go ahead. 

Once it runs fully to completion, it will show an output IP address. Open the IP address in your web browser and you should see Hello World! from our user data. 

Congratulations, you have successfully created a new webserver on AWS using Pulumi and Python. 

The above code can be found here https://github.com/gordonmurray/pulumi_webserver 

Summary 

In this post we have been introduced to Infrastructure as code, the process of creating and updating cloud-based infrastructure items using code.  

We learned of the advantages of this approach, such as committing our infrastructure code to a source control repository such as GitHub, so code changes can be peer reviewed before deploying changes into production. 

We looked at the four popular IaC tools including AWS CloudFormation, Hashicorp’s Terraform, Redhat’s Ansible, and Pulumi as four different technologies to use to create and update cloud infrastructure and gained some practical experience by using each tool to create a simple webserver on AWS. 

In a future post, we will explore a number of deployment tools to help us to deploy changes into production. Deployment tools such as Travis or Jenkins can help us to perform repetitive steps and allow us to work as part of a team to review changes and deploy safely. 

Building infrastructure locally on a Private Cloud

In this post we will cover a number of ways to create a private cloud. The term private cloud is a term used to describe a number of servers owned by an individual or a business to run their own software systems. Its resources are not usually shared with other businesses like in a ‘public cloud’ such as AWS, Google or Azure.

A private cloud could be as small as a single server in your home, or multiple servers rented from a third party internet provider. The approaches we will cover in this post can be applied to a PC or laptop to configure it to work as a server or applied to dedicated servers in a server room at your place of work. In our examples, we will focus on working on a single server.

We will configure a PC into a working server running Ubuntu Server. This home server is an ideal place to create and learn many of the items we cover in this post. If you are applying any of the steps we take in this post to a PC in your home, don’t forget to back up any important information first, as these steps will replace whatever is currently on the PC.

In this post, we will cover the following topics:

  • Manually provisioning physical servers
  • Managing provisioning of physical servers using maas.io
  • Deploying virtual machines on your physical servers using KVM
  • Deploying Docker containers on your physical servers
  • Managing containers using Portainer.io

Manually provisioning physical servers

Provisioning a server is the process of installing an operating system to configure the server, to make it ready for production use. Creating and managing physical servers manually can be a time-consuming process and includes the following steps:

  1. The first step is to download an ISO (ISO 9660) file for the operating system you are planning to install. An ISO file is an archive of files intended to be copied to a CD or Universal Serial Bus (USB) device.
  2. The next step is to copy this ISO image to a CD or USB drive.
  3. Once a CD or USB device is ready, it is plugged in to a target computer to install your new operating system.
  4. When installing the operating system, some manual input is needed to answer questions related to drive partitions, users, software packages to install and more.

In our examples, we will install Ubuntu Server 20.04 LTS and its ISO can be downloaded here https://ubuntu.com/download/server

If you prefer to use a different version of Linux or use a Microsoft Windows operating system the steps will be roughly the same to install.

If you want to work with an alternative to Ubuntu, this site lists a number of different Linux distributions and links to download their ISO files: https://www.linuxlookup.com/linux_iso

If you want to work with Windows, Microsoft provide a number of ISOs available here https://www.microsoft.com/en-gb/software-download for different versions of their operating systems.

In the sections that follow, we will work with Ubuntu Server. We will follow a number of steps to create a boot-able USB device. This USB device will install the Ubuntu Server and make it ready for our next steps of creating a web server.

Creating a bootable Ubuntu Server USB

The Ubuntu.com website provides up to date guides on creating a bootable USB device for installing Ubuntu Server. There is a suitable guide if you are using Linux, Windows or Mac, with detailed step by step instructions and screenshots to show you how to download an ISO, copy it to a USB device that can be used as a bootable device to install Ubuntu:

Create a bootable USB divide from Ubuntu: https://ubuntu.com/tutorials/tutorial-create-a-usb-stick-on-ubuntu#1-overview

Create a bootable USB device from a Mac: https://ubuntu.com/tutorials/tutorial-create-a-usb-stick-on-macos#1-overview

Create a bootable USB device from Windows: https://ubuntu.com/tutorials/tutorial-create-a-usb-stick-on-windows#10-installation-complete

Now that we have created a bootable USB drive, our next step is to use a laptop, desktop or dedicated server to boot from this disk and create a working server in our next section.

Installing Ubuntu Server

Once you have created your bootable USB or CD from the previous section, you are ready to begin installing Ubuntu Server on a system that you would like to turn into a server.

Ubuntu.com also provides a detailed step by step guide on installing Ubuntu Server here: https://ubuntu.com/tutorials/tutorial-install-ubuntu-server#1-overview

As you follow the steps, you will see that the steps are straightforward and help you to get a working Ubuntu Server system. However, the steps are also time consuming to follow, especially if it was necessary for you to create 10 or 100 servers for your business. Later in this post we will look at ways to speed up this process.

Once Ubuntu Server is provisioned, your server is ready to configure. The configuration steps you need to perform will depend on the purpose of your server. We can’t include all potential configurations in one book so we will focus on one popular use of a server, which is to configure it to be a webserver to host one or more websites.

In our next section, we will configure a LAMP (Linux, Apache, MySQL PHP) web server

Configuring a LAMP (Linux, Apache, MySQL, PHP) web server

In our previous sections we created a bootable USB or CD and installed Ubuntu Server on our server. In this section we will perform a number of steps to configure this server into a working web server to host one or more websites. Using the following commands in our terminal, we can

sudo apt update

This apt update command uses apt, Ubuntu’s package manager. The command updates apt to make it aware of the newest versions of packages and their dependencies.

sudo apt install php apache2 mariadb-server -y

This command uses apt to install PHP, a popular scripting language. Apache2 a popular web server and finally Maria DB, a popular relational database. The -y command is short for ‘Yes’, to continue to perform the download without asking to confirm.

If you open the IP address of your new server in your web browser, you will see the default homepage of Apache web server.

If you are unsure how to see the IP address of your server, you can run the following command on the server to determine its current IP address:

ifconfig | grep inet

You will see an output similar to the following:

inet 192.168.1.132 netmask 255.255.255.0 broadcast 192.168.1.255

The first IP address of 192.168.1.132 is the IP address of your server. Open this IP address in your web browser to see the Apache homepage.

Congratulations, you have provisioned a new server to use Ubuntu Server and configured it to become a web server.

If you wanted to begin to host a website from this address, you could need to upload files to the server to a folder located at /var/www/html/.

In this section, we created one single web server. In our next section we will use MAAS, a tool which will help us to provision multiple servers over a network connection.

Managing provisioning of physical servers using maas.io

Maas.io is a service developed by Canonical, the company that is also responsible for creating Ubuntu. Machine-As-A-Service (MAAS) once installed, operates as a controller to help you to deploy an operating system onto one or more servers over your network.

Earlier we covered the process of downloading ISO images, copying them to a USB device and then taking the time to create a working server from scratch, one at a time. With Maas, the process is a little different. The Maas software can be installed on a server acting as a controller or on your own PC or laptop. Once installed, you will be asked a small number of questions such as a region name and then Maas is ready to provision servers.

If you are using Ubuntu already on your PC, MAAS is quick and easy to install as shown in the following command:

sudo snap install maas

sudo maas init

MAAS will ask a short number of questions to set itself up, as follows:

  1. Choose all and accept the default URL
  2. Add a username, a password and add an email address
  3. Skip the import key step for now unless you have a key to import

Once the steps have been completed, you can open up the Maas dashboard at:

http://127.0.0.1:5240/MAAS/#

Once you open the Maas dashboard, you will be asked one or two remaining questions.

  • Set a ‘region’ name such as ‘home’ or ‘office’
  • Choose a DNS forwarder, such as Googles DNS addresses of 8.8.8.8 8.8.4.4 as shown on the screen
  • Choose one or more ISO images you would like Mass to download to its own library to install on a server later ( this process can be slow depending on your internet connection, so for now choose just 1 ISO such as Ubuntu 20.04)
  • Press Update Selection to save the Image choices
  • Press continue
  • Import your SSH public key
    • Choose Upload
    • Then paste in your local public key
    • If you are unsure of your local public key, open a terminal and type the following cat ~/.ssh/id_rsa.pub
    • Copy the output and paste it into to field called Public key in the maas set up screen
    • Press Import
    • Press Go to dashboard

You now have maas.io installed. If you have access to other physical servers in your home or place of work you can boot them up and instead of using a CD or USB with an ISO installed, you can set the machine to boot over the local network (sometimes called Preboot Execution Environment (PXE).

Once you have one or more physical servers start up in this way, they will connect to the maas controller. The servers will appear in the maas user interface, ready to provision. This process allows you to quickly and easily install a number of different operating systems on your physical devices locally or remotely. Once a server has been provisioned with an operating system, it is ready to be configured in a webserver for any purpose.

An alternative service that operates in a similar way to Maas.io is Digital Rebar, available at https://rebar.digital/. Digtal Rebar provides the same functionality as Maas.io however it also

supports orchestration tools such as Ansible and Terraform which we will cover in our next post for creating and provisioning cloud infrastructure

Deploying virtual machines on your physical servers using KVM

KVM stands for Kernel-based virtual machine. In the previous section, we provisioned a physical server and configured it into a LAMP web server. While this process works well, it is also possible to further “divide” a server into multiple different virtual servers that can even run different operating systems.

KVM is just one method of creating virtual machines and in this section we will look at how to create one or more virtual machines using KVM on your existing server.

Use the following command in your terminal to install KVM and some necessary dependencies:

sudo apt install qemu-kvm libvirt-bin virtinst bridge-utils cpu-checker

Once it has been installed, you can verify it using the command:

kvm-ok

You will see an output similar to the following:

INFO: /dev/kvm exists

KVM acceleration can be used

Now we can create a new VM using the following:

virt-install \
–name ubuntu \
–description “Our first virtual machine” \
–ram 1024 \
–disk path=/var/lib/libvirt/images/ubuntu-1.img,size=10 \
–vcpus 2 \
–virt-type kvm \
–os-type linux \
–os-variant ubuntu18.04 \
–graphics none \
–location ‘http://archive.ubuntu.com/ubuntu/dists/bionic/main/installer-amd64/’ \
–extra-args console=ttyS0

When you arrive at the Software selection screen, make sure to install OpenSSH Server. This will allow you to connect to the VM using SSH when it is finished installing as shown in the following image:

Figure 3.1- Software selection

To see a list of virtual machines running on your server, you can use:

virsh list

You will see an output similar to the following:

Id Name State

—————————————————-

1 ubuntu running

To SSH into the server, we need to find out its IP address. Each virtual machine has some XML metadata which includes the details of the virtual machine, such as its name, its description and most importantly its network information.

To export the meta data of a virtual machine, use the following command, using the ‘Name’ of the virtual machine from the virsh list command:

virsh dumpxml ubuntu

(where ubuntu is the name of the virtual machine)

This will output some XML to your screen. The XML can be over 100 lines long, however the part we are interested is in the Mac Address as shown in the following snippet:

<interface type=’user’>
<mac address=’52:54:00:66:65:51’/>
<model type=’virtio’/>
<alias name=’net0’/>
<address type=’pci’ domain=’0x0000′ bus=’0x01′ slot=’0x00′ function=’0x0’/>
</interface>

Within the XML, look for a field called ‘MAC address’. A MAC address is an identification number that identifies each device on a network. The Mac address will look something like 52:54:00:65:27:b7

We can now use another command called arp to see the IP address of the device using that Mac address. ARP stands for Address Resolution Protocol. This protocol is used by network nodes to match IP addresses to MAC addresses.

In your terminal, use the following command to look up the MAC address:

arp -an | grep 52:54:00:65:27:b7

This command will show an output similar to:

? (192.168.122.23) at 52:54:00:65:27:b7 [ether] on virbr0

In the round brackets, you will see an IP address similar to 192.168.122.23, this is the IP address of the virtual machine you have created.

To SSH into the server (make sure you installed OpenSSH when installing Ubuntu on the virtual machine) you can use the following command:

Ssh {username}@192.168.122.23 ( where username is that name you supplied when installing Ubuntu)

You will be prompted for the password you provided during the installation also and then you will be logged in successfully to your virtual machine.

Once logged in, you could run the following commands like earlier in this post to turn the new VM into a webserver:

Sudo apt update
sudo apt install php apache2 mariadb-server -y

The virsh program is the main interface for managing virtual machines. The program can be used to create, pause, and shutdown machines. If you would like to see more commands use:

virsh –help

While this process is useful to make a virtual machine, it is still a slow process as you will need to answer questions during the installation. In a later section named Creating custom images for KVM using Packer we will look at creating customized virtual machine images. This means performing the configuration first and then booting up a completed virtual machine, ready to use.

Cloning your virtual machine

Creating multiple physical or virtual machines takes time. If you have taken the time to create and configure a virtual machine in the last section, you can use this as a template for creating additional virtual machines.

Cloning a virtual machine is the process of taking a copy of an existing virtual machine and creating a new copy of it with a different name, leaving the original virtual machine in place unchanged.

Before cloning an existing virtual machine, it needs to be shut down first using the following command:

virsh shutdown ubuntu

To clone, or copy of the existing virtual machine, use the following command to copy the ubuntu virtual machine and create a new ubuntu-’ machine:

sudo virt-clone –original ubuntu –name ubuntu-2 –file /var/lib/libvirt/images/ubuntu-2.img

You will see the following output:

Allocating ‘ubuntu-2.img’ | 10 GB 00:00:25

Clone ‘ubuntu-2’ created successfully.

You can now start your old and new VMs using the following commands:

virsh start ubuntu

virsh start ubuntu-2

You can see both machines running using:

Virsh list

The number of virtual machines you can run depends on the available CPU, memory and disk space on your server.

A useful command to keep an eye on your available disk space is:

df -h

This will show the size used and available space on each drive on your server. The name for your disk drive might change but it is likely to be called /dev/sda1.

A useful command-line program to monitor CPU and memory is called htop, which can be installed using:

Sudo apt install htop -y

And then running htop with:

htop

The following image shows,

Figure 3.2 –

This will show you a live screen of running processes, memory and CPU being used. You can press q to quit.

Now that we have created a virtual machine, is it available to use and learn. In our next section, we will show how to delete the virtual machines if you no longer need them.

Deleting your virtual machine

If you would like to delete your virtual machines, you can use the following command:

virsh destroy ubuntu

(where ubuntu is the name of your virtual machine from using virsh list)

Once the virtual machine is removed, its image will remain in the /var/lib/libvirt/images/ folder and might take up a lot of space depending on the size of the disk you told it to create.

To save on disk space, delete the image using the following command:

sudo rm /var/lib/libvirt/images/ubuntu.img

This section showed us how to create virtual machine images using commands in our terminal. In our next section we will use Packer, a piece of software developed by Hashicorp to create images which can be saved and used repeatedly.

Creating custom images for KVM using Packer

Packer is a tool created by a company called Hashicorp. It is a tool that allows you to describe a server in a JSON format and then build that server into a number of different image types. A packer file could create an AWS EC2 AMI, while another packer file could create a QEMU Copy On Write version 2 (QCOW2) image, suitable for virtual machines using KVM which we will use here.

Packer is a tool that can help to automate building of virtual machine images. Its JSON files can be committed to a repository such as Github as part of working towards Infrastructure as Code, an approach to infrastructure development which we will cover in the next post.

To install packer, follow the instructions on the Packer website for instructions suitable for Windows, Linux or Mac:

https://www.packer.io/intro/getting-started/install.html

Once packer has been installed, we can begin to create a new packer file. In the following example we will create a CentOs image instead of an Ubuntu image mainly as it is a smaller file and easier to learn, compared to an Ubuntu packer build. You can see an example of an Ubuntu packer build in the source code for this post.

Create a new file called centos.json, with the following content:

{
“variables”:{
“centos_password”:”centos”,
“version”:”2003″
},

This variables block allows you to declare some variables to use later in the build. In this case we are creating a password that will be used to SSH into the server if needed and a version of CentOs to download and use.

Next, we add the following

“builders”:[
{
“Vm_name”:”centos-packer.qcow2″,
“output_directory”:”output-centos”,
“iso_urls”:[
“iso/CentOS-7-x86_64-Minimal-{{ user `version` }}.iso”,
“http://mirror.de.leaseweb.net/centos/7/isos/x86_64/CentOS-7-x86_64-Minimal-{{ user `version` }}.iso”
],

The Vm_name field is used to control the name of the output file when the build is finished and the output directory controls the folder to use. In this case it is a qcow2 file that is supported by KVM.

The fields labelled iso_* are used to describe where to get the ISO file to install and a checksum va

Next, we add:

“iso_checksum_url”:”http://mirror.de.leaseweb.net/centos/7/isos/x86_64/sha256sum.txt”,
“iso_checksum_type”:”sha256″,
“iso_target_path”:”iso”,
“ssh_username”:”centos”,
“ssh_password”:”{{ user `centos_password` }}”,
“Ssh_wait_timeout”:”20m”,

The ssh_* fields controls the username, password and timeout values to use when connecting to the virtual machine. The SSH timeout here needs to be enough time for Packer to SSH into the server and configure it to make it 10 minutes at least.

Next, we add

“http_directory”:”http”,
“boot_command”:[
“<tab> text ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ks.cfg<enter><wait>”
],
“boot_wait”:”2s”,
“shutdown_command”:”echo ‘{{ user `centos_password` }}’ | sudo -S /sbin/halt -h -p”,

The http_directory is a folder that is shared with the host operating system and the virtual machine being created. The boot_command uploads a kickstart file called ks.cfg to this folder so that the virtual machine can read it. The kickstart file is a configuration file that can have answers already written out for questions that are asked during installing the operating system to automate the process. The boot_wait allows a few seconds for the VM to start up and the shutdown command gives the command to shut down the machine to complete the build.

Next, add:

“type”:”qemu”,
“headless”:true,
“memory”:2048,
“Cpus”:2

The type key has a value of qemu. It is the type needed to create a virtual machine image that KVM can use. Memory and CPU describe the memory and CPU given to the virtual machine. The key ‘Headless’ with a value of ‘‘true’ is needed to run the build without a visual display on a desktop. We can finish the file by closing the brackets as follows:

}
]
}

The full file content can be viewed on GitHub at: https://github.com/gordonmurray/learn_sysops/blob/master/chapter_3/packer/centos/centos.json

Once the file is created, we can perform the following command to validate the file:

Packer validate centos.json

And once the file is ok, we can run the following command to build the image:

Packer build centos.json

The build will take a few minutes to run as it downloads the ISO, verifies it, installs CentOS and answers the questions for you.

Once the image has been built successfully using Packer, we can launch the virtual machine using the following command, similar to creating a new virtual machine earlier in this post:

virt-install –import \
–name centos \
–description “Our centos virtual machine” \
–ram 1024 \
–disk centos-packer.qcow2,format=qcow2,bus=virtio \
–vcpus 2 \
–virt-type kvm \
–os-type linux \
–os-variant=rhel7.5 \
–graphics none \
–check all=off \

Once the virtual machine has been created, you will be able to see it by running virsh list again to see the new virtual machine.

If you SSH into the virtual machine you’ll see the operating system is already installed and ready to configure.

If you would like to take the packer build further, you can add optional ‘provisioners’. We will cover Provisioners in more detail in a later post called Infrastructure as Code. Provisioners can be used to run commands such as installing software during the packer build so that you don’t need to install the same software after you launch the virtual machine.

In our next section, we will leave virtual machines behind and look at an alternative, Docker containers.

Deploying Docker containers on your physical servers

So far we have looked at creating a new physical server and configuring it to become a webserver. We then learned how to create virtual machines and create custom images using Packer. Another popular approach to deploy software to a server is to use Docker containers. When compared to virtual machines, containers can be smaller in disk size, easier to create and take up less resources to run, making them a great option for packaging and deploying modern software and use server resources more efficiently.

In an earlier post in ‘Local Containers’, we introduced the concept of Docker containers and in this section we will build and deploy a working container on to our server.

In your editor, create a new file called Dockerfile with the following contents:

FROM ubuntu:18.04
RUN apt-get update && apt-get install -y apache2
ENV APACHE_RUN_USER www-data
ENV APACHE_RUN_GROUP www-data
ENV APACHE_LOG_DIR /var/log/apache2
ENV APACHE_RUN_DIR /var/www/html/
EXPOSE 8080
CMD [“/usr/sbin/apache2”, “-D”, “FOREGROUND”]

To build an image from this Dockerfile, use the following command in your terminal:

docker build –pull –rm -f “Dockerfile” -t ubuntu:latest “Docker”

This process will take a few minutes to build an image. Once the process has finished, you can see a list of available Docker images using the command:

Docker images

You will see an output similar to:

REPOSITORY TAG IMAGE ID CREATED SIZE

ubuntu latest fbdb9b18aa5a 22 seconds ago 189MB

To run this image so it becomes a working container, use the following command:

docker run –rm -d -p 8080:8080/tcp ubuntu:latest

To see a list of running containers use the following command:

Docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

f608b6672159 ubuntu:latest “/usr/sbin/apache2 -…” 8 seconds ago Up 6 seconds 0.0.0.0:8080->8080/tcp happy_cerf

To connect to the running container use the following command:

docker exec -it a9fa0e376b95 /bin/bash

(where a9fa0e376b95 is the container ID from the last command).

Earlier in this post we created a virtual machine and then configured it to become a web server. With Containers this process usually is to do the work ‘up front’ in the Dockerfile, so that when an image is built, it already has all items configured, ready to run, so that you don’t need to connect to the container and install any software.

If you would like to take Docker further, it is possible to store your Docker images in Dockerhub, which is very similar to storing source code in Github. Dockerhub accounts are free to set up and you can tag and push your images to store in Dockerhub and deploy to your servers using a number of deployment practices and third parties tools that we will cover later in this book.

Docker Compose

Docker compose is a process whereby a number of different container images are described in a single file, often called a docker-compose.yml. When instructed, Docker Compose will build or retrieve all necessary images and start up a number of running containers.

Docker compose can be installed locally, on a virtual machine or run in a cloud environment such as AWS.

Using docker compose allows you to coordinate running a number of container images. Often when developing a web application you will need a number of different elements such as your main web application, a database, some storage for files or even a cache such as redis.

Docker allows you to create your own container images for all of those needs and docker compose allows you to instruct your server to run each of those container images so they can all run and communicate with one another to make a complete web application.

In our next example we will use Docker Compose to create 2 containers, a web server and a MariaDB database server.

First we need to install docker compose on linux for which you can use the command:

sudo apt install docker-compose

For installing Docker compose on other operating systems such as Mac or windows, you can find instructions on the Docker website at https://docs.docker.com/compose/install/

Create a file called docker-compose in the same folder as your earlier Dockerfile, with the following content:

Docker-compose.yml

version: ‘3’
services:
web:
build: .
ports:
– “8080:80”
database:
image: “mariadb:latest”

In this file we are declaring 2 services, web and database. In web we are telling docker compose to build the Dockerfile in the current directory by using the value of `.`.

In the database section, we are telling docker to get the latest version of MariaDB, a popular relational database.

Next we can perform the following command to start our containers:

docker-compose -f docker-compose.yml up -d –build

Once the web service has been built and the database service downloaded, you will see an output similar to:

Creating docker_database_1 … done

Creating docker_web_1 … done

We can view our running containers using docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

5e7b1e9990d7 mariadb:latest “docker-entrypoint.s…” 2 minutes ago Up 2 minutes 3306/tcp docker_database_1

cdeab94c8b24 docker_web “/usr/sbin/apache2 -…” 7 minutes ago Up 7 minutes 8080/tcp, 0.0.0.0:8080->80/tcp docker_web_1

In this section, we used docker-compose to create 2 images. We declared a web image and instructed docker to build the Dockerfile in the current directory.

We then created a second database image and instructed docker to get a premade mariadb iamge from Dockerhub.

In our next section we will look at Portainer, a piece of software that provides a user-friendly interface for managing docker images.

Managing containers using Portainer.io

Portainer is a web application that you can run to provide an easy to use user interface for your containers and related actions.

If you are running on a linux system is it very quick to install and get up and running using Portainer using 2 Docker commands:

docker volume create portainer_data

docker run -d -p 8000:8000 -p 9000:9000 –name=portainer –restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer

The first command creates a volume to store portainer data and the second command runs the portainer container. Once installed, open your web browser at http://localhost:9000, you will be prompted to supply a password and you will be then logged in.

If your containers are still running from earlier in this book, you will see them listed in the ‘Containers’ section of Portainer.

To gain more experience running servers from containers, Portainer can provide an ideal user interface to get used to creating and deploying working environments.

Portainer provides App Templates which is a list of common services such as databases or blog services, so that you can deploy easily to learn or to be used by a web application.

Summary

In this post we covered a number of ways to create and use a physical server. These same tools and approaches can be used to create simple home server for learning, or used in a work environment to create servers and configure them to provide useful services to staff and customers.

In a future post, we will explore other tools and technologies such as Ansible that can be used to further develop or automate some of the steps we have taken in this post to configure a new server, to create virtual machines and even create and deploy docker containers.

Further reading

local development options and an introduction to Amazon Web Services (AWS)

This post is an introduction to developing infrastructure in both a local development environment such as a laptop or desktop and also developing in a public cloud environment. Local development is an ideal location for learning, usually without any additional costs when compared to developing in the cloud environment.

We will learn about some tools for local and cloud based infrastructure development. These tools will help you in later series as well as provide a number of options for your own development needs.

We’re going to cover the following main topics:

  • Local development using Virtual machines
  • Local development using Docker Containers
  • An introduction to AWS
  • Creating a web server using EC2 instances
  • Creating a web server using Elastic Container Service (ECS)
  • Creating a web server using Elastic Kubernetes Service (EKS)

Learning infrastructure development 

Developing infrastructure on cloud service providers such as AWS can be fun. However, If you are not careful, leaving infrastructure items running can cost a lot.

Cloud service providers such as AWS and Google Cloud offer a free tier that are ideal for learning.  Many services are free to use for a period of time, and some services are completely free to use at all times if you remain  under certain usage limits.

For more information on the services that AWS and Google offer as part of their free tier, see their the following pages:

https://aws.amazon.com/free/

https://cloud.google.com/free

To help avoid some of those financial costs, we will look at some cost saving steps later in this series.

In the meantime, there are several ways to simulate a production environment and develop locally on your laptop or PC.

Local development

Local installation is a process you’re probably already familiar with. It is the process of installing or writing software on your PC or laptop. Installing software locally can be a great way to learn and gain experience with different software languages and tools.

Local installation of a software language such as PHP, Python or Go can be convenient. It allows you to write code in your editor and compile or run your software locally which has a convenient and quick development cycle as well as being easier to learn.

Local development can also lead to challenges when working as part of a team or on multiple software projects. Your local setup might be different to your coworkers, potentially slowing you down as you work together.

As you progress you may work with many different pieces of software or tools, often with different versions of those tools. You might like to have multiple different versions of a programming language installed to use in different projects, this is where virtual machines or Containers can become very useful.

Local Virtual machines

A number of tools exist including VM Workstation, VirtualBox or Parallels which allow you to create a virtual copy of an operating system within your existing local machine. The specific tool you use will depend on your local machine and the items you are developing.

If you are running on a Mac, you can create a Windows or Linux based virtual machine (VM) for example.

Creating a virtual machine comes with some overhead, such as using up additional space on your machine as well as both CPU and memory. However it will provide you with a consistent and repeatable environment that you develop and test on.

Virtual machines can be stopped when not in use and started up again when you need them. They can also be copied and shared with coworkers so everyone is using the same environment when working within a team.

Since VMs can be copied, they can provide an ideal place to learn how to work within a new operating system. You can create a new Virtual Machine, set up an operating system you want to use and create a copy of it. While you learn the new OS, if you break it in any way, you can quickly revert to the earlier copy and keep going.

Tools such as Vagrant, which runs on Windows, Mac and Linux machines allow you to describe your chosen Virtual Machine through code. Pre made ‘boxes’ can be downloaded and then started up on your device so you don’t need to install and configure the operating system within the VM yourself.

Once you have Vagrant installed, an example of a simple Vagrantfile is below. Running vagrant up in the same folder as the vagrantfile will create an Ubuntu 18.04 Bionic 64 bit Virtual machine locally in seconds.

Vagrant.configure(“2”) do |config|

config.vm.box = “bionic64”

config.vm.hostname = “bionic.box”

config.vm.network :private_network, ip: “192.168.0.10”

end

Virtual machines run a full operating system within the virtual environment, so this can take a toll on your local machine. Depending on the power of your device you might be able to run 2 or 3 virtual machines at the same time before your device slows down too much to be usable. This is when Containers can come in handy.

Local containers

Containers are much smaller than virtual machines, in both disk space usage and also in terms of memory and CPU usage when they are running.

Containers do not contain a copy of an operating system, they contain the application and dependencies needed to run.

Since Containers use less resources than Virtual Machines, you can run many more Containers at once than VMs on your devices.

Tools such as Docker, rkt and LXC allow you to create custom containers as well as run them on your local machine.

Once you have Docker installed, an example of a simple Dockerfile is below. This will download an Ubuntu image, install and run Apache web server and make a web page available on port 8080

FROM ubuntu:18.04

RUN apt-get update && apt-get install -y apache2

ENV APACHE_RUN_USER www-data

ENV APACHE_RUN_GROUP www-data

ENV APACHE_LOG_DIR /var/log/apache2

ENV APACHE_RUN_DIR /var/www/html/

EXPOSE 8080

CMD [“/usr/sbin/apache2”, “-D”, “FOREGROUND”]

The above example shows a typical Dockerfile. Later in this post we will build our own Dockerfile to run our own application and cover the steps used within the file in detail

Introduction to AWS 

In this section we will cover some of the most fundamental resources available within AWS that you will need to be familiar with when you are building any infrastructure on AWS.

The AWS Management Console

The AWS management console, accessible at https://aws.amazon.com/console/ is the main user interface for AWS services. Once you log in, you’ll see icons representing all of the many services that AWS offers.

If you don’t already have an account, you can sign up at no cost. Many AWS services have a ‘free tier’, meaning they are completely free to use for your first year within certain limits. There are also services that can be used at no cost beyond the first year if you remain under certain limits specific to that service. One example is AWS Simple Notification Service (SNS) which is a great messaging service that can connect events in one AWS service to another, it can even be used to send SMS messages. You can use up to 1 million Publishes per month at no cost.

AWS Simple Email Service (SES) is another great service, used to allow applications to send emails. You can send over 60,000 emails per month from SES at no cost.

These free services make AWS an ideal starting point for learning about infrastructure. You can see some detailed information on the free services that AWS offer here on their website https://aws.amazon.com/free

In this post, we will explore Elastic Compute Cloud (EC2), Elastic Container Service (ECS) and Elastic Kubernetes Service (EKS). Each of these services can be used to host software you have developed and are part of the ‘Compute’ section of Amazon Web Services. These services can also work with other AWS services to build and deploy changes to your infrastructure and software.

Virtual Private Cloud (VPC)

A Virtual Private Cloud (VPC) is a service provided by AWS to contain and control networking in your AWS infrastructure.

Within your AWS Account you can have one or more VPCs and if you have signed up to AWS for the first time, AWS will create a default VPC for you.

A VPC will allow you to control traffic and the security within your network of services.

When creating a VPC you can define a custom Classless Inter-Domain Routing (CIDR) block. This means you can define a range of IP addresses for your AWS network that items within the VPC will use. For example a range of IPs such as 10.0.0.0 will mean that any EC2 instances you create will have an internal IP address of 10.0.0.x. A VPC also contains other networking resources, such as Subnets, Route tables and Security groups to manage web traffic. We will cover each of those items now.

Subnets

You can use Subnets to separate these IP ranges further. If you plan on deploying a large number of different items such as EC2 or RDS instances, you can use Subnets to define different ranges for each service to use. EC2 instances could use a range of 10.0.0.x while your RDS instances could use a separate range of 10.0.x.x. This allows you to separate your instances and avoid potential IP address conflicts.

Subnets can also be public facing or private. Public subnets allow items such as E2 instances to be publicly accessible. A good example of an EC2 instance in a public subnet is a standard web server, where you want your users to be able to access these servers. A subnet can be public facing using an Internet Gateway which we will cover later in this post.

With a Private subnet, you can keep instances within the subnet from being contacted by the outside world. This is ideal for keeping sensitive items such as databases within RDS private and only accessible from within the network. A private subnet is a subnet with no Internet Gateway attached. If you need to allow your Private subnet to connect to the internet, you can use a Nat Gateway.

You will need to allow your EC2 instances which are in a different subnet to connect to your RDS instances, you can do this using Route Tables which we will cover next.

Route Tables

While subnets can define a range of IP addresses used by your services, Route tables allow your subnets to talk to each other, or not depending on your infrastructure needs.

A route table can have many subnets attached though a subnet can be connected to only 1 route table.

Internet Gateways

An Internet Gateway connected to a subnet allows the subnet to become a public subnet. This allows items within the subnet to be accessible from the broader internet.

Security Groups

With a VPC in place, along with one or more subnets connected to a route table and an Internet Gateway, your EC2 instances are now exposed to the Internet. A Security Group allows you to control the specific traffic that is allowed to contact your EC2 instance.

For example, creating a Rule in your Security group to allow IP addresses in the range of 0.0.0.0./32 on Port 80 will allow all users on the internet to contact your EC2 instances on port 80. This can be a common rule as it allows public internet traffic to talk to a web server such as Apache or Nginx on your web server.

Other common Security Group rules include opening port 22 to your home or office IP address. This allows you to SSH into your Linux server.

Windows users might like to open port 3389 to their home or office IP address so allow an RDP connection to the server.

EC2 instances

EC2 is probably the item you will use the most on AWS. EC2 instances are virtual machines you can start and stop at any time. EC2 instances can be one of many different types from Linux to Windows based virtual machines with 1 CPU up to over 100 CPUs. As we look at other AWS services later on such as ECS or EKS, you will find that those services rely on EC2 instances.

EC2 instances are where you will deploy your web application if you are developing one. New EC2 instances can be launched from within the AWS console. You will be asked a number of questions including the instance type, size, any additional volumes to attach to the instance, their disk size, the security group(s) to apply to the instance, and a Key Pair.

Once you have made your decisions, the EC2 instance will usually launch within a minute or two.

Once created, EC2 instances can be stopped, copied or scaled in size. A copy of an EC2 instance is called an Amazon Machine Image (AMI) and AMIs can be used to back up a running EC2 instance or copy the image to other AWS regions or accounts to make more identical EC2 instances.

EC2 Instance types

AWS provides a number of different instance types. The types provided are for different needs.

  • Instance Types of T2, T3, M4 and M5 are General purpose instances. They contain a balance of both CPU and available memory. Useful for most workloads.
  • Instance Types of C4 or C5 are Compute (CPU) instances. They usually contain more CPUs with moderate increases in memory for each different size.
  • Instance Types of R4, R5, X1 are memory-optimized instances. The amount of available memory in each instance size will double while the CPU will increase moderately.
  • Instance Types of P2, P3, G4, G5 are instances with additional GPUs. These instances are ideal for crunching data.
  • Instance Types of I, D, H are instances with plenty of storage.

Each of these instances come with different underlying hardware, per hour pricing as well as different availability and pricing per geographical AWS region.

If you are starting a new EC2 instance for the first time and you are unsure which Type to use, one of T3 Types can be a good start. They are low powered and low cost instance types and you can grow from there if you need more resources.

For more information on current instance types https://aws.amazon.com/ec2/instance-types/

AWS also provide a Simple Monthly Calculator to help you to estimate the costs of various AWS services before you create them https://calculator.s3.amazonaws.com/index.html

Key Pairs

When you are creating a new EC2 instance, as well as the additional questions mentioned earlier, you will also be asked to create or use an existing Key Pair.

A Key Pair is in the form of a file with a .pem extension. Pem stands for Privacy Enhanced Mail (PEM), a format originally intended for use with email.

This .pem file is the public key half of a public/private key pair. AWS can create one for you, or you can use the Key Pair page within the EC2 section of the AWS Console to import your existing public key and store it as a .pem file. A .pem file can be used when creating any new EC2 instances in your account.

Once you have created an EC2 instance with a .pem key, don’t lose that file as you will need it to connect to the instance later.

Spot Instances

Spot instances are just like any other EC2 instances, however they are instance types that are low cost and depend on the AWS market demand and availability.

Spot instances cost considerably less than a standard EC2 instance however the instances can be short lived and so might terminate at any time, with only a small 2 minute warning.

When creating a new EC2 instance you can select to create a Spot instance. The Console will show you the current price and you can then enter the amount you are willing to pay per hour. As long as your bid amount is the same or higher than the current price of the Spot instance, your instance will start up and remain available to use. As the AWS market for availability changes over time, the price of the instance may increase. If it goes above your big price, your instance will terminate.

Depending on the instance type, your bid price and the availability of the instance type, spot instances can persist for weeks or months without termination.

Reserved Instances

With Reserved Instances (RIs), you can create EC2 instances as normal, or work with EC2 instances that are already running.

Reserved Instances are a commitment to run one or more EC2 instances for 1, 2 or 3 year periods in return for a lower hourly cost for the instances.

You don’t need to make any changes to an EC2 instance for it to become a Reserved Instance. The Reserved Instance is a separate process that works alongside EC2 instances.

If you have a number of EC2 instances running you can apply for the Reserved Instances for those EC2 instances. The Reserved Instances must be the same as the EC2 instances that are running to cover them effectively.

For example, if you have an M5.large EC2 instance running, you can then apply for the same M5.large Reserved Instance. This will lower your hourly running cost for that instance.

When creating a new Reserved Instance, you have a number of options. You can choose to reserve for 1, 2 or 3 year periods. A 3 year RI will be cheaper to run that a 1 Year RI.

The thing to watch out for with Reserved Instances is that if you set up an EC2 instance and a Reserved Instance effectively, it will save you money. However if you change your EC2 instance over time so that the EC2 instance and RI no longer match, then you will end up paying for both. You will pay for the normal EC2 instance cost, plus the additional cost of the instance type described in the RI.

If the number of EC2 instances you have in your account will stay the same over a long period of time without any change to their size, then RI’s can be very effective items to use.

The ‘My Billing Dashboard’ section of the AWS control panel can help you to see how much of your EC2 instances are covered by Reserved Instances.

Creating a basic web server on EC2

With the information we have learned about AWS and EC2 above, we can now go ahead and create a web server hosting a ‘Hello World’ page on a new EC2 instance.

The following steps are all possible using the AWS console however the AWS console interface may change over time and the steps will no longer be valid.

Instead, we will use the AWS Command Line Interface (CLI) in our terminal to create an EC2 instance and its dependencies in a small few steps.

Installing the AWS CLI

The AWS CLI is supported on Windows, Mac and Linux machines. A list of instructions for each one is available on the AWS website at: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html

The examples in this section assume you are working from a linux or mac machine.

Installing JQ

JQ is a command line tool for parsing JSON content. When using the AWS CLI, a JSON output is one of the output responses that is supported. Often the output can be verbose and jq is an ideal tool to help parse that JSON response.

JQ is supported on Windows, Mac and Linux machines. A list of instructions for each one is available on the JQ website at: https://stedolan.github.io/jq/download/

Creating our webserver using the AWS CLI

The following commands use the AWS CLI to create the resources we will need on AWS to start an EC2 instance and upload some files to it, to act as a web server.

The following steps assume there is an existing VPC and at least 1 subnet which are available by default in a new AWS account:

Use your terminal to perform the following:

  • First, create a new file called index.html with the content of ‘Hello World!’

echo “Hello World!” > index.html

  • Next, use the following command to determine your current public IP address and take note of it. You will need this later to tell the Security group to allow you access the server.

curl https://www.canihazip.com/s

  • Use the following command to list the ID of one more VPC’s in your account and take note of the output:

aws ec2 describe-vpcs –region us-east-1 –query ‘Vpcs[*].VpcId’

  • Use the following command to list the subnet IDs available in the VPC chosen from the previous step:

aws ec2 describe-subnets –region us-east-1 –query ‘Subnets[*].SubnetId’ –filters “Name=vpc-id,Values=YOUR_VPC_ID”

  • Create a new security group that will be used by our EC2 instance:

aws ec2 create-security-group –region us-east-1 –group-name example-security-group –description “For an example EC2 instance”

  • Add a rule to the security group to allow public traffic to access the default Apache webserver port 80:

aws ec2 authorize-security-group-ingress –region us-east-1 –group-name example-security-group –to-port 80 –ip-protocol tcp –cidr-ip 0.0.0.0/0 –from-port 80

  • Add another rule to allow your local machines IP address to connect to the EC2 instance later via SSH:

aws ec2 authorize-security-group-ingress –region us-east-1 –group-name example-security-group –to-port 22 –ip-protocol tcp –cidr-ip YOUR_IP_ADDRESS_HERE/32 –from-port 22

  • Query the security group that was created in the earlier step to get its security group ID and note its output:

aws ec2 describe-security-groups –region us-east-1 –group-names example-security-group –query ‘SecurityGroups[*].[GroupId]’ –output text

  • Create a new Key Pair in your AWS Account and save a local copy of it too as a file called example.pem:

aws ec2 create-key-pair –region us-east-1 –key-name example –query ‘KeyMaterial’ –output text > example.pem

  • Create the new EC2 instance based off an Ubuntu 18.04 LTS instance. Substitute the Security group ID and Subnet ID from the earlier steps:

aws ec2 run-instances –region us-east-1 –image-id ami-04b9e92b5572fa0d1 –count 1 –instance-type t2.nano –key-name example –security-group-ids SECURITY_GROUP_ID_HERE –subnet-id SUBNET_ID_HERE  –tag-specifications ‘ResourceType=instance,Tags=[{Key=Name,Value=example}]’

  • Query the EC2 instance we just created to get its Public DNS name so we can connect to it later:

aws ec2 describe-instances –filter “Name=tag:Name,Values=example” –region us-east-1 –query ‘Reservations[].Instances[].[PublicDnsName]’ –output text | head -2 | tail -1

  • Set owner read and write permissions to the key pair file we created earlier:

sudo chmod 600 example.pem

  • Use SCP to copy a local index.html page to the home folder on the new EC2 instance, substitute the Public DNS address from the earlier step:

scp -i example.pem index.html ubuntu@PUBLIC_DNS_HERE:/home/ubuntu

  • Use SSH to connect to the server and perform a couple of steps. Perform an apt update to update the system, install apache2 web server and move the index.html file to the webserver root folder at /var/www/html/

ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no ubuntu@PUBLIC_DNS_HERE -i example.pem “sudo apt update && sudo apt install apache2 -y && sudo mv /home/ubuntu/index.html /var/www/html/index.html”

Now, open your web browser and open up the public dns name from the earlier step and you should see the contents of your index.html file.

Congratulations, you have created your first web server, running on an EC2 instance in a VPC on AWS!

Removing your Webserver

The items we created earlier such as the EC2 instance will cost you money as you leave them running. The following steps will help you to delete the items to clean up.

  • List EC2 instances in your account and note the Instance ID:

aws ec2 describe-instances –filter “Name=tag:Name,Values=example” –region eu-west-1 –query ‘Reservations[].Instances[].[InstanceId]’ –output text

  • Delete the EC2 instance based on its instance ID from the earlier step:

aws ec2 terminate-instances –region us-east-1 –instance-ids INSTANCE_ID_HERE

  • Delete the security group we created earlier. It will remove the 2 security group rules we added earlier too:

aws ec2 delete-security-group –region us-east-1 –group-name example-security-group

  • Delete the key pair we created earlier from the AWS Account:

aws ec2 delete-key-pair –region us-east-1 –key-name example

  • Remove the local copy of the key pair:

rm example.pem

The full list of steps combined in one file is visible here on Github https://gist.github.com/gordonmurray/4e6541f689205db3eb9bddc17a95fb1c

Using the AWS CLI is very powerful, however you will see that if you try to run some of those commands again, you will receive errors as the items already exist. Later in this series, we will look at provisioning tools such as Ansible and Terraform. These tools can also create AWS items such as EC2 instances, if run multiple times they will either create or confirm the instance exists, rather than causing an error.

In the following sections we will look at a number of additional services such as AWS ECS and AWS EKS which use EC2 instances, as examples of AWS services that can be used to run our web services.

Introduction to AWS ECS 

AWS ECS stands for Elastic Container Service. It is a service AWS provides for running applications within containers, instead of apps deployed directly to EC2 instances.

With ECS, you do not need to worry about the underlying EC2 instances directly. The ECS service will deploy your tasks to one or more available EC2 instances to keep your service running.

If you are starting out with deploying apps within containers, ECS can be a great service to use. Kubernetes is another popular container orchestration tool that we will cover later in this series.

ECS comes with some of its own terminology that we’ll need to know to continue:

  • Cluster  – A group of EC2 instances. When an EC2 instance launches, a piece of software called the ecs-agent runs on the server and registers this new  instance to ECS Cluster.
  • Task Definition – This is information that describes how a docker container should launch. It contains settings like exposed port, docker image, CPU, memory requirement, command to run and environmental variables.
  • Task – This is a running container with the settings defined in the Task Definition. It is like a running version of a Task Definition.
  • Service  – Defines a Task that needs to be kept running.

For more information, take a look at AWS’s ECS site at https://aws.amazon.com/ecs/

Container registry

When you have created one or more Docker Containers to run your different applications, you need somewhere to store those Container images.

In the same way that a service like Github can store your code so you can collaborate with a team, Container Image Registries provide a similar server for storing and sharing your container images.

Elastic Container Registry (ECR) is a Docker container registry provided by AWS to store and share your container images. For more information on ECR, see https://aws.amazon.com/ecr/. Docker Hub (https://hub.docker.com/) is another popular Docker container image repository.

Building a Docker Image

Before you begin, you will need a Dockerhub account, https://hub.docker.com/ and you will also need Docker installed on your machine.

To begin using ECS, we will need to create an application that runs within a Container. Once we have created this application, we need to build an image and push it to Dockerhub, so that the ECS service can contact Dockerhub to pull the image to run it in the cluster.

To create a Docker image, we use a file called a Dockerfile. Within this file we can describe a base image to build on top of, then add our own files and build the image.

  • First, let’s create a simple Hello World PHP page that we will add to the Dockerfile in the next step. Create a file called index.php in a folder called src.

<?php

//src/index.php

Echo “Hello World from Docker!”;

  • Then, create a file called Dockerfile in a new folder with the following content:

FROM php:7.2-apache

COPY src/ /var/www/html

EXPOSE 8080

RUN usermod -u 1000 www-data

RUN chown -R www-data:www-data /var/www/html

Lets cover each of the sentences used in this Dockerfile to explain their role in creating a container:

The FROM command tells Docker to build an image starting with the image called php:7.2-apache. This is a pre-built image with PHP version 7.2 and Apache installed. You can start from any image you like and install new items on top of it, this image helps keep the Dockerfile small for demo purposes.

The COPY command tells Docker to copy all contents from the /src folder on your local machine, which should just be the index.php file so far, into the /var/www/hml/ folder in the Image

The EXPOSE command tells the Docker image to listen on port 8080

The usermod command changes the user ID of the www-data user to 1000. UIDs begin with 1000 and UIDs below 1000 are reserved for system use.

The chown command tells Docker to change the /var/www/html to be owned by the www-data user and group so that Apache has permission to manage files in that folder

  • You can build an image from this Dockerfile using the following command:

sudo docker build –rm -f “Dockerfile” -t example:latest .

This command calls docker build to create a new image from the Dockerfile. It is called ‘example’ and tagged with ‘latest’. Tagging allows you to give your images different names, often used for versioning your images.

  • You can see a list of your images using the command:

Sudo docker images

This will show you a list of any images you have built. These are image files and don’t represent a running app yet. To run your image, use the following:

sudo docker run –name=example -it –rm -p 8080:80 example:latest

  • You can confirm your container is running using:

Sudo docker ps

Open your web browser and open http://localhost:8080 and you should see the content from the /src/index.php file!

If you wanted to develop this app further, you would need to update your files in the src/ folder, then build the image again using the docker build command, then run it again using the docker run command. For ongoing local development, this can be a great way to work, though building and running repeatedly can be time consuming. Instead, it is possible to ‘mount’ your local development work from the src/ folder into an already running container and any time you change a file on your local machine, it will be available within the container and visible in your browser straight away.

Now that we have a small app built as an image, our next step is to push this image to Dockerhub so that later on ECS can pull it and run it within the ECS cluster.

  • Once you have created your Dockerhub account, you will need to log in to push your images. You will also need to create a Personal Access Token to use as your password when logging in. You can do this by logging in to Dockerhub, going to your Profile and opening the Security section.

docker login –username=YOUR_DOCKERHUB_USERNAME

  • Next you will need to tag your image for Dockerbub:

sudo docker tag example {YOUR_DOCKERHUB_USERNAME}/example

  • Finally, push your image to Dockerhub using the following:

sudo docker push {YOUR_DOCKERHUB_USERNAME}/example

Once pushed, you should be able to see your saved Image on Dockerhub at https://hub.docker.com/

Our app is now an image located at {YOUR_DOCKERHUB_USERNAME}/example, we will use this location address in our next section when creating an ECS cluster and deploying our app to the cluster.

Creating a basic web service on ECS

The following steps assume you have an AWS account with the AWS CLI installed locally. We will perform several steps to create an ECS cluster.

  • Use the following command to list the ID of one more VPCs in your account and take note of the output

aws ec2 describe-vpcs –region us-east-1 –query ‘Vpcs[*].VpcId’

  • Use the following command to list the subnet IDs available in the VPC chosen from the previous step

aws ec2 describe-subnets –region us-east-1 –query ‘Subnets[*].SubnetId’ –filters “Name=vpc-id,Values=YOUR_VPC_ID”

  • Create a key pair for accessing the EC2 instances.

aws ec2 create-key-pair –key-name example –query ‘KeyMaterial’ –output text > example.pem  –region us-east-1

  • Next, create two security groups. One will be for the EC2 instances and the other will be the Application Load Balancer (ALB)

aws ec2 create-security-group –region us-east-1 –group-name instances-sg –description “For EC2 instances”

aws ec2 create-security-group –region us-east-1 –group-name alb-sg –description “For ALBs”

  • When the security groups are created, they will output their ID after the command has run successfully, in the format of sg-xxxxxxxx where the xxx’s are unique to the security group. With the security groups created, we will add a small number of rules to the security groups to allow the public in on port 80 for web access and a rule to allow to ALB and the EC2 instances to communicate with one another on any TCP port from 1 to 65535

aws ec2 authorize-security-group-ingress –region us-east-1 –group-name instances-sg –to-port 80 –ip-protocol tcp –cidr-ip 0.0.0.0/0 –from-port 80

aws ec2 authorize-security-group-ingress –region us-east-1 –group-name alb-sg –to-port 80 –ip-protocol tcp –cidr-ip 0.0.0.0/0 –from-port 80

aws ec2 authorize-security-group-ingress –group-name instances-sg –protocol tcp –port 1-65535 –source-group alb-sg

  • The next step is to create the Application Load Balancer. An application load balancer is able to receive web traffic and send it to different targets, such as a target for EC2 instances that will be used by our ECS cluster. You will need to fill in your application security group ID from the earlier step. You will also need to provide at least 2 subnet IDs. Use 2 subnets from the earlier step that listed all available subnets.

aws elbv2 create-load-balancer  –region us-east-1 –name example-load-balancer –security-groups ${LB_SG} –subnets {SUBNET_ID_1} {SUBNET_ID_2}

  • Now that the load balancer has been created, we will need to know its Amazon Resource Name (ARN). This is a name given to AWS Resources. Store this ARN for later.

aws elbv2 describe-load-balancers | grep LoadBalancerArn

  • Next, we will create a Target Group. A target group is usually a group of one or more EC2 instances. In this case, we will leave the target group empty and ECS will populate it later when we add a service.

aws elbv2 create-target-group –name example-targets –protocol HTTP –port 80 –target-type instance –vpc-id {VPC_ID}

  • Just like with the load balancer, now that the target group has been created, we need to query it to get its ARN. Save this ARN for later also.

aws elbv2 describe-target-groups | grep TargetGroupArn

  • Next we will create a listener on the load balancer. A listener tells the load balancer to listen for traffic on one or more ports and send that traffic to a target group. You will need to substitute in your load balancer ARN and your Target group ARN from earlier.

aws elbv2 create-listener –load-balancer-arn {LB_ARN} –protocol HTTP –port 80    –default-actions Type=forward,TargetGroupArn={TG_ARN}

  • Now we’re getting to the interesting steps, next we will create the cluster called ‘example’

aws ecs create-cluster –cluster-name example

  • Next we will create a file to tell an EC2 instance the name of our ECS cluster so it can join it when it starts up. Create a file called data.txt with the following content

#!/bin/bash

echo ECS_CLUSTER=example > /etc/ecs/ecs.config

With the cluster started, we will create one EC2 instance to join the cluster, if you’d like to start more, then change the ‘count 1’ value to a number you are happy with.

This is simply an EC2 instance that will become part of the ECS Cluster. It is using an ECS optimized AMI which has a docker and the ecs-agent running on it. We are passing some user data from a file called data.txt to the instance which is a way to pass some commands to an EC2 instance when it is starting up for the first time.

You will need to substitute in a subnet ID and your instance security group we created earlier

aws ec2 run-instances –image-id ami-097e3d1cdb541f43e –count 1 –instance-type t2.micro –key-name example subnet-id {A_SUBNET_ID} –security-group-ids {INSTANCE_SG} –iam-instance-profile Arn=arn:aws:iam::016230046494:instance-profile/ecsInstanceRole –user-data file://data.txt

Our EC2 instance will take a few seconds to start up. Next we will create a Task Definition. This describes how a docker container should launch. It contains settings like ports, docker image, cpu, memory, command to run and env variables.

Create a file called `definition.json’ with the content below. This is where we tell ECS to use our Docker image that we created earlier. You will need to substitute in your Dockerhub username.

{

“family”: “hello-world”,
“containerDefinitions”: [
{
“name”: “php-hello-world”,
“image”: “{YOUR_DOCKERHUB_USERNAME}/example:latest”,
“cpu”: 100,
“command”: [],
“memory”: 100,
“essential”: true,
“portMappings”: [
{
“containerPort”: 8080,
“protocol”: “tcp”
}
]
}
]
}

  • Register the task definition

aws ecs register-task-definition –cli-input-json file://definition.json

Next we will create a Service. A service is responsible for maintaining the tasks we have described in the task definition.

Create a file called service.json with the following content. You will need to substitute in the target group ARN from earlier.

{
“cluster”: “example”,
“serviceName”: “example-service”,
“taskDefinition”: “hello-world”,
“loadBalancers”: [
{
“targetGroupArn”: “{YOUR_TARGET_GROUP_ARN}”,
“containerName”: “php-hello-world”,
“containerPort”: 8080
}
],
“desiredCount”: 1,
“role”: “ecsServiceRole”
}

Next we will use the AWS cli command to create this new service using the file we just created.

aws ecs create-service –cli-input-json file://service.json

We are nearly there. Earlier we created a load balancer which will receive the traffic and pass it onto the target group. This target group has been updated by the Service we created in the last step. Our next step is to query the Public DNS name of the load balancer so that we can access it. You will need to substitute the load balancer ARN from earlier.

aws elbv2 describe-load-balancers –load-balancer-arns {LB_ARN} | grep DNSName

Open the DNSName that is returned in your web browser to see your running container output!

The above steps are available in a single script here: https://gist.github.com/gordonmurray/259eb3c52e66188ea4b0e3b420a6ccd8

Removing your ECS cluster

Keeping a cluster running can add up in cost if you have several EC2 instances and load balancers running. The following steps will help you to remove the steps from above. Later in the series, we will work on some ways to reduce your cluster costs with Spot instances.

  • Get the instance ID of the EC2 instance to delete it in the next step

aws ec2 describe-instances –filter “Name=tag:Name,Values=example” –region us-east-1 –query ‘Reservations[].Instances[].[InstanceId]’ –output text

  • Delete the EC2 instance

aws ec2 terminate-instances –region us-east-1 –instance-ids ${INSTANCE_ID}

  • Get the Amazon Resource Name (ARN) of the load balancer

aws elbv2 describe-load-balancers –region us-east-1 –names “example-load-balancer” –query ‘LoadBalancers[*].[LoadBalancerArn]’  –output text

  • Get the ARN of the listener on the load balancer

aws elbv2 describe-listeners –load-balancer-arn ${ALB_ARN} –query ‘Listeners[*].[ListenerArn]’ –region us-east-1 –output text

  • Delete the Listener based on its ARN

aws elbv2 delete-listener –listener-arn ${ALB_LISTENER_ARN} –region us-east-1

  • Delete the Load balancer based on its ARN

aws elbv2 delete-load-balancer –load-balancer-arn ${ALB_ARN} –region us-east-1

  • Get the ARN of the Target group

aws elbv2 describe-target-groups –region us-east-1 –names “example-targets” –query ‘TargetGroups[*].[TargetGroupArn]’ –output text

  • Delete the target group based on its ARN

aws elbv2 delete-target-group –target-group-arn ${TG_ARN}

  • Delete the security groups

aws ec2 delete-security-group –region us-east-1 –group-name instances-sg

aws ec2 delete-security-group –region us-east-1 –group-name alb-sg

  • Scale the running service down to 0 so it can be deleted

aws ecs update-service –cluster example –service my-service –desired-count 0 –region us-east-1

  • Delete the ECS service

aws ecs delete-service –cluster example –service my-service

  • Delete the ECS cluster

aws ecs delete-cluster –cluster example –region us-east-1

  • Delete the key pair and the local pem key file

aws ec2 delete-key-pair –region us-east-1 –key-name example

rm example.pem

The above steps are available in a cleanup script here: https://gist.github.com/gordonmurray/259eb3c52e66188ea4b0e3b420a6ccd8

In this section we learned how to build and use Docker images to package our web application. We created a Dockerhub account and learned how to store our container images in a Container registry so that those images could be used by services such as ECS or Kubernetes. We then used the AWS command Line (CLI) to create an Elastic Container Service (ECS) cluster and add a single EC2 instance so our application would have available resources to run.

Our next step was to create a Task definition and a Service to describe our image and its requirements to run and deploy our app to run on ECS. mOnce our app was running we were able to query its DNS name and view the output in a web browser.

Our final optional step was to clean up and remove any resources we created so that we wouldn’t have any items still running afterwards.

Amazon’s ECS is one of several great options for deploying container based web applications into production. In our next section we will look at another AWS based solution to also deploy container based web applications, Amazon’s Elastic Kubernetes Services (EKS), a hosted version of Kubernetes.

Introduction to AWS EKS

Kubernetes is an open source container orchestration tool, developed by Google. It is similar to AWS ECS which we covered earlier, in that Kubernetes can be used to run and scale Docker containers spread over a number of host instances.

AWS Elastic Kubernetes Service (EKS) is AWS’s hosted version of Kubernetes. Using a hosted version of Kubernetes such as EKS can save time and can be easier to learn when compared to setting up your own clusters manually, even if you are using Infrastructure as code tools such as Terraform.

Kubernetes has a large community and a growing number of tools. This series can’t cover all the aspects of it, in this section we will cover creating an EKS cluster and deploying an app to the cluster.

Before we begin, you will need to install a number of command line tools

  • eksctl – eksctl is a shortened version of EKS Control. It was originally developed by Weaveworks and has become the official command line tool for EKS. The following page will show you how to install eks locally for your system https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html
  • kubectl – kubectl stands for Kubernetes Control. kubectl is a powerful command line tool. If you work with Kubernetes, you will likely work with this tool a lot. kubectl can be installed independently of eksctl, however it is better to install AWS’s version of kubectl for an easier set up process to work with eksctl. The following page will show you how to install kubectl https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html
  • Helm – Helm is a package manager for Kubernetes. Helm is not strictly required to work with Kubernetes, however it is a very powerful and practical package manager to help you to package your containers, deploy to Kuberntes, and roll back to earlier versions if there are problems. This following page will show you how to install Helm https://docs.aws.amazon.com/eks/latest/userguide/helm.html

With eksctl installed, we can now create an EKS cluster in one very easy command:

eksctl create cluster \
–name example \
–version 1.14 \
–region us-east-1 \
–nodegroup-name example-workers \
–node-type t3.medium \
–nodes 3 \
–nodes-min 1 \
–nodes-max 4 \
–managed

The above command will take around 10 minutes to complete. It will create a small cluster using version 1.14 of Kubernetes with 3 nodes. Nodes are the name that Kubernetes gives to EC2 instances. You can change the number of nodes if you wish to make a smaller or bigger cluster.

Once the command has completed, you can begin using kubectl to see if the cluster is ready to use:

The following command will show you how many nodes (EC2 instances) your cluster is using

kubectl get nodes

The following command will show any services deployed to the cluster. It won’t include a lot right now since it’s a new cluster

kubectl get services

If you’d like to know more kubectl commands, you can use kubectl help at any time to get a list of available commands. The Kubernetes site also offers a useful cheat sheet of common commands https://kubernetes.io/docs/reference/kubectl/cheatsheet/

Helm charts

Your cluster is now up and running and our next step is to deploy some apps into the cluster using Helm.

In the same way that we can store and share code on Github and Docker container images on Dockerhub, we also have Helm Hub for sharing helm packages known as Charts, located at https://hub.helm.sh/

  • To begin installing Helm charts, we need to first add a chart repository, a common char repository is the official Helm chart repo

helm repo add stable https://kubernetes-charts.storage.googleapis.com/

  • Once the stable repository is added, you can search it using:

helm search repo stable

  • To install a chart, first make sure your chart repositories are up to date and then install a chart, in this case we will install a mysql chart

helm search repo stable

helm install stable/mysql –generate-name

The generate-name parameter will generate a name for the running service so you can identify it.

Once the chart is installed, it should produce some useful information on screen with connection information.

Try running kubectl get services again to see the mysql service is running.

Congratulations, you have just deployed your first application to a Kubernetes cluster using Helm!

Creating your first custom Helm chart

Earlier when we created our ECS cluster, we built a simple app container image and pushed it to Dockerhb. In this section, we will use the same app and push it to our kubernetes cluster.

  • Go to a new directly and use the following command to create a new Helm chart for your app

helm create php-hello-world

This will create a directory structure for you with a number of files.

  1. In the values.yml file, update the repository to: YOUR_DOCKERHUB_USERNAME/example
  2. In the chart.yml, change the appVersion to : appVersion: latest
  3. We can now deploy the app using the following helm command:

helm install php-hello-world ./php-hello-world –set service.type=NodePort

When the app has been deployed, Helm will give you a couple of commands to issue. This will provide you with the IP address and port where your app has been deployed to, so you can visit it in your web browser.

You have now deployed your first custom app to Kubernetes using Helm!

Cleaning up

If you would like to remove your application from the cluster, you can delete the app using the following

helm delete php-hello-world

If you would like to remove your EKS cluster fully, you can run the following eksctl command.

eksctl delete cluster example

In this section we built on the container knowledge of the previous section. We learned to use new tools including eksctl, kubectl and Helm to create a Kubernetes cluster on AWS. We learned how to use Helm to search for existing premade charts such as mySQL and we learned to package our own container image using a custom Helm chart to deploy our application into our Kubernetes cluster.

Finally, we used a number of commands to remove the EKS items we have created in this section as an optional step to clean up instead of having services running that could cost us money if left running accidentally.

Some recommended reading

  • Docker: Up & Running: Shipping Reliable Containers in Production – Book by Sean P. Kane
  • Docker Deep Dive – by Nigel Poulton
  • The Kubernetes Book – by Nigel Poulton

An introduction to SysOps and DevOps

The role of a SysOps engineer and the kind of work they perform can vary from company to company. For the context of this piece, we will use the title of SysOps Engineer to describe the role of a person that is hired by a company to develop and maintain the infrastructure that is required for software products to run in a production environment.

A person in a SysOps role is responsible for a system and keep all of its various elements running smoothly. This can include anything from a single server, to multiple servers, databases and any third-party dependencies that a system relies upon to operate.

There are many variations to the title, System Administrators, SysAdmins, Ops, SysOps, Operations, and more recently System Reliability engineers (SRE), a title created at Google.

Whichever title you decide to choose in your career, the aim of this content is to give you an introduction to the terms and practices of a SysOps engineer as well as some practical experience with several tools and technologies to help you start your SysOps career.

What is DevOps?

DevOps means a cross department cooperation between Developers and SysOps engineers. It is a cultural approach within a company intended to drive efficiency and reduce problems when developing and deploying software to production.

The term DevOps is intended to describe an approach to work. For developers to develop code with the complete system in mind as much as possible or to work closely with the SysOps team during each phase of ongoing development.

By working together, software can be developed quickly and safely.

Before DevOps, software development often followed a Waterfall method of development. This approach broke development up into logical stages such as gathering requirements, planning, designing, developing, and testing including security, deployment, and maintenance.

This process was rigid and slow. The stages were linear and it wasn’t possible to move on to the next stage until the previous stage was completed. Changes to the plan once it was in progress were either blocked or slowed down the project. It isn’t always possible to know all of the requirements at the beginning of the process and there was no going back, resulting in failed or dropped projects.

Agile methods became popular to change this approach. To create a minimum working software product quickly to satisfy customer’s needs and to iterate again and again in short development cycles to add new features or to fix any bugs.

The danger of not taking a DevOps approach, where developers and SysOps stick to their own particular areas of work, or silos, can result in a poorly developed product or service.

If not adopting a DevOps approach to work, developer and SysOps teams may end up working in different directions or at different speeds without considering each other’s work.
Some examples of working in silos:

A developer might work on some code without thinking of things that can happen in a production environment. Events such as DNS issues, or a database failover that don’t happen when developing locally can happen in production and a developers code may not be ready to handle it in a graceful manner.

A development team might deploy some great new features into production for its users. These new updates might include new database queries being performed, putting additional pressure on a busy database. This could impact negatively on performance for users and could have been avoided if communicated in advance with the team responsible for looking after the database infrastructure.

A SysOps team working in isolation without communicating with development teams may work to improve a system and make changes to the infrastructures in ways that the developers aren’t expecting. As a simple example, upgrading a database to the latest security patched version, while any existing apps are expecting the previous database version. This is the kind of scenario that can arise when teams don’t work together for a common goal.

For example, this can result in putting the database under pressure, large process lists, increasing CPU or memory, and begin setting off alarms for the SysOps team to react to.

Your customers might be impacted by this, they may just have gained access to new features in your software and suddenly it has stopped working or performance has dropped significantly.

The solution to it might be an easy one, to scale horizontally by increasing your available database replicas or to upgrade the database itself to handle the work, though those options are all reactive and you’re likely doing it at peak traffic which isn’t an ideal time to find out about it.

A development team and a SysOps team working closely can prevent these kinds of issues from occurring by being aware of each other’s work, communicating often, and keeping the customers in mind.

Becoming a SysOps Engineer

A SysOps engineer can often start their careers as a software developer and grow into a SysOps role out of necessity.

A modern website or web application is more than just software. When you’re working alone, or as part of a small team, you may be the only person that can develop the software as well as any items it needs, such as file storage, the ability to send emails, or a database to store user data.

Creating a database, or managing space for users to upload files may be outside of your comfort zone as a developer though these items are necessary for your software to work well and be useful for its users.

The aim of this content is to guide you on how to create and manage items like this. You may feel overwhelmed in the beginning, or even along the way, as there is a lot to learn, though try to take one thing at a time. If something is failing, try to become a mini expert in it. Search for information on it, back it up, take notes and try to fix it one change at a time.

Some tasks you will need to perform as a SysOps engineer can be basic, such as freeing up some disk space mentioned earlier on a server that is running out of space. Try to automate tasks like this, or better still prevent it from happening.

A developer might turn to Google, blog posts or books on how to add these items to their software as the need arises.

The server you are working on could be a Linux-based or a Windows-based machine, so over time you’ll need to become familiar with some basic commands on each system.
In later chapters, we’ll learn about some AWS specific ways to solve common problems such as creating servers, load balancing, databases, and backups.

Things you’ll need to know as a SysOps engineer:

  • Being a developer – While it isn’t essential to be a Developer before moving to a SysOps position, any development experience you already have will certainly help. As a result of being a developer, you may already be familiar with many of the tools and concepts such as git, testing, compiling, deploying changes, and more, which we will explore in more detail later in this site. If you don’t have a developer background, a good place to start would be to have a side project of your own, where you learn a new software language and build a small application start to finish
  • Command line – Command line usage is common in a SysOps role. You’ll need to be familiar with the different command lines offered by different systems. Linux-based systems often use a command line called Bash, while Windows systems use a Dos-based command prompt or PowerShell. Using the command line can be a real time saver compared to performing any actions manually in your file manager for example. Command line actions can also be shared with coworkers or added to scripts to automate several tasks.
  • SSH Public and Private keys – SSH keys comprise of two files. The first is a public key file and the second is a private key file to complete the pair. SSH keys are a great way to connect to remote servers to deploy your code. You’ll always keep your private key to yourself and the public key can be shared with a remote server to authenticate your users. Once you have your keys in place, connecting to a remote server is as easy as ssh username@remotehost which connects you to your remote server. Many services such as Git and Deployment tools use keys which we’ll cover in more detail later.
  • Databases – There are many database options available out there, from file-based SQLite to multi-master relational databases such as mySQL and noSQL databases such as MongoDB.
    • You don’t need to know of every kind of database. The database(s) you use will likely be dictated by the software you or your organization are developing. If you are already familiar with one kind of database such as mySQL, it would be worth the time to look into other databases to know their strengths and weaknesses and to know how to add/remove users, set user permissions, and perform day-to-day actions such as creating/altering tables/collections.
    • A brief introduction to different types of common databases:
      • Relational databases such as mySQL or MS SQL are ideal for structured data such as user details, online shopping product details, blog posts, and most other common application uses.
      • NoSQL databases such as MongoDB or AWS DynamoDB are ideal for unstructured data such as JSON data. NoSQL databases can be ideal if you are taking in third-party data that you don’t control or data that may change its structure over time without warning.
      • In-memory databases such as Redis are fast key/value databases. They can help improve the performance of an app by keeping frequently accessed data readily available rather than your software connecting to a potentially busy database as often.
      • Graph databases such as Neo4J are ideal for storing data where relationships matter. Ideal for showing social network relationships or complex data networks.
      • Time series databases (TSDBs) are ideal for storing data where the sequence is important, such as application logs.
      • Multi-model databases are becoming more common. Database offerings such as ArangoDB or OrientDB can support two or more different data models in one, which can be ideal if the app you are developing needs more than one type of data storage.
  • Version Control – Using a version control system such as Git is a way to store the source code of your software. Changes are committed to a repository and other members of the team can access and modify the same files where permitted.
    • Even if you are the only developer on a project, using a version control system is still a good idea to use to get familiar with some branching models for when you work in a team later on. Some useful terminology when using Git:
      • A commit is the process of adding some changes to the local copy of your repository.
      • A push is the process of pushing one or more commits from your local machine to the central repository – so your coworkers can see the work for example.
      • A pull is the process of pulling any changes from the main repository to your local machine.
      • A branch is a way to take a copy of the source code where you can work on making changes or bug fixes and when you are ready to submit your work after one or more commits, you can push your work to the branch. This branch becomes visible to team members to contribute to or comment on while the original codebase remains unchanged.
      • A PR is a Pull Request. This is a process where you can request that your branch is merged into the main code so that your work becomes part of the main application.
      • A fork is similar to a branch, however a branch often only exists for a short time before being merged or being dropped. A fork will usually mean that a developer is taking a full copy of the source code for their own use, to make significant additional changes that they may never want to merge back into the main source code.
    • Popular version control services include Github, Gitlab, and Bitbucket. These services are an ideal starting point for deploying software changes to production, which we will cover in more detail later.
  • Web servers – If you are developing a web application, you’ll want to put it online somewhere so that other people can use it. You have a number of options for running your application. You can self-host it on a physical server you own, or use a cloud-based virtual server. Or you can package your app to run within a container or use a serverless approach. Later in the series, we’ll give an intro to each approach with practical examples so that you can choose an approach that works for you and the software you develop.
    • Using any approach will give you a great experience in developing, deploying, and managing your web application.
  • Deployments – A deployment is the process of taking completed code and sending it to one or more destination servers for users to use. There are many different types of deployments, from single file changes, to more detailed blue/green deployments. Blue/Green deployments are where new servers are created on demand, user traffic is diverted to the new servers when ready and the old servers are removed afterwards.
    • The specific deployment process you use will depend on the software you are developing and the deployment destination, such as virtual servers, containers or serverless functions. The following is a list of common deployment tools:
      • FTP/SFTP – This is the process of sending files, often one at a time to a destination server. FTP stands for File Transfer Protocol and commonly uses port 21 to communicate with the destination server. SFTP is Secure File Transfer protocol and often communicates with port 22 on the destination server. FTP is an older protocol and it is generally not recommended to use as there are some newer more secure approaches available with better features.
      • Rsync – Rsync stands for Remote Sync. It is a tool for syncing files from one server to another. It is useful as it will only send the files that have changed locally to the destination server. It can also maintain any preexisting file permissions, compress files, and show progress as it syncs files.
      • SCP – CP short for Copy, is a common command line facility to copy files and folders locally. SCP is Security Copy that can copy files to a remote server.
      • Git Pull – This is the process of ‘pulling’ files inward, from a remote git repository on to the current server. Some local processes on the server will need to trigger the pull step. This process isn’t generally recommended as it requires a copy of Git to be installed locally, along with a .git folder which can often contain sensitive information if the server is compromised.
      • AWS API/CLI – The AWS CLI is a command line interface to many AWS services. The AWS API or CLI can be used to send files to AWS S3, a popular choice for static frontend websites or applications that talk to an independently deployed backend API.

Exploring other forms of Ops

DevSecOps

In the same way that DevOps aims to bring Development + Operations closer together to improve the development process, DevSecOps aims to bring security to be an integral part of all stages in the development life cycle.

Instead of leaving Security measures to the end of the development, or omitting security measures entirely, security is to be kept in mind at each stage of development, from writing code, to compiling binaries, to deployment and ongoing maintenance.

While writing code, developers can add security-related tests such as SQL injection attempts, or CSRF attempts in to their unit tests.

While building a server, an Ops team can choose to harden their base server or container images using Center for Internet Security (CIS) or other standards. Hardening a base image in this way to remove common vulnerabilities can be automated using tools such as Ansible or Chef, which we cover later in this series.

When deployed to production, the Ops team could continue to scan and monitor the applications.

AWS provides a number of security-related services including AWS Guard Duty, a service which can perform ongoing scans of your infrastructure for malicious activity.

GitOps

GitOps is a concept introduced by a company called WeaveWork. GitOps is similar to the concept of Infrastructure as Code ( IaC). With IaC, your infrastructure is created and managed using software such as Terraform or CloudFormation. This code can be treated just like any other application code and added to the source code repository such as Github. The same workflows often employed in software development, such as branches, PRs and merges can be applied to IaC resulting in the desired changes being deployed to update your infrastructure in place.

Whereas IaC is aimed at the entire infrastructure, GitOps tends to be aimed at container orchestration tools such as Kubernetes. In a Kubernetes cluster, software can be deployed and managed using a range of tools starting with KubeCTL.

Deployments can be performed independently of one another and it isn’t always clear at a high level what applications have been deployed at a given time and what version of those applications are currently running within the cluster.

With a GitOps approach, information on the deployments that should be running within the cluster are written in a declarative format such as a YAML file and committed to the repository. When new software needs to be deployed to the cluster, the files in the repository are updated which triggers a background process to bring the cluster up to date to match the desired outcome.
If additional applications are deployed to the cluster independently of the gitops approach, such as deploying directly using KubeCTL, then the cluster will automatically be reverted back to the version described within the repository.

This approach gives a very clear summary of the applications and versions currently deployed to the cluster. If one or more applications need to be deployed to the cluster, a developer can update the repository and deploy a new app or roll back to an earlier version using a PR. This approach can also facilitate rebuilding a damaged cluster, or deploying a copy of the same applications to a different cluster, used for testing or staging for example.

ChatOps

ChatOps is the name given to the process of bringing development related conversations, actions, and workflows into a Chat application used at your organization.

Also known as conversation-driven development, the title may have originated with Github, they have a number of ChatOps related videos on YouTube that are worth watching.

There are a number of reasons why a company might like to use a ChatOps approach. A team can work together developing a software product and keep any conversations in one place, visible to the entire team, instead of spread out over emails or other sources.

ChatOps also works well for any teams with remote team members, or those in different time zones. Team members can sign in when they begin their day and see any conversations, actions or alerts that may have occurred while they were offline.
While performing any day-to-day development work, events such as commits or deployments can be sent in to the same chat rooms so team members can see the ongoing development. Any important events such as a failed deployment can be seen straight away for the team to respond to.

By performing actions in a shared chat room, your team can share information easily as they work. New team members will be able to see the actions and perform the same steps on their own.

If you are using AWS, a service called Short Notification Service (SNS) is a great resource for receiving notifications for events that occur within AWS. Events such as an RDS instance reboot can send an event to SNS and in turn sends you a notification via Email, SMS, or Webook. Events can also be sent to AWS ChatBot, a service that can receive the event information, process it, and send it on to popular Chat apps such as Slack.

One item to be aware of when automating actions within a Chat application is to keep actions secure. If you have automated an action such as rebooting a server, you might want to limit that action to a specific room or set of users to avoid misuse.

NoOps

NoOps (No Operations) is the idea that things can be so automated that a company won’t need dedicated SysOps roles.

NoOps could be thought of as the end result of a successful move to a DevOps culture. This is one where developers can build, deploy, and monitor their software through automation and self-service with no need for a sysops team.

NoOps is a worthwhile goal. If you can automate things to a degree that makes your work or a developer’s work easier, you’ll be in a great place with a lot of experience under your belt.

Don’t be afraid of automation

Some recommended reading

  • “The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win”, by Gene Kim, Kevin Behr and George Spafford
  • “The Unicorn Project: A Novel about Developers, Digital Disruption, and Thriving in the Age of Data”, by Gene Kim

Setting up a Salt cluster on DigitalOcean using Terraform

I use Ansible to provision or automate infrastructure tasks whenever I can. I wanted to try out Salt, from SaltStack to see how it compared. I read about its ability to push commands out to multiple servers very quickly and I wanted to try it for myself.

To gain some practical experience using Salt I used Packer and Terraform to create a Salt Master and several Salt Minions on DigitalOcean.com. This post is a guide on how to create a small Salt cluster of your own to begin learning Salt. The end result will be 1 Salt Master and 3 Salt Minions, running on the smallest DigitalOcean droplets, so you can safely experiment without a large cost.

Packer and Terraform are both great tools from Hashicorp. Packer allows you to describe a server in code, customize it and build it as an Image that you can store on cloud providers such as DigitalOcean or AWS for later use.

There are 2 overall steps to get the Salt cluster up and running. First I use Packer to create 2 base Images, a Salt master and a Minion and then I use Terraform to create the Droplets on DigitalOcean based on those images.

Packer is doing the work of provisioning the server Images ahead of time, so once the servers are running, you won’t need to SSH in to them to configure them unless there are problems.

To follow this guide, you will need:
* Install Terraform https://www.terraform.io/intro/getting-started/install.html
* Install Packer https://www.packer.io/intro/getting-started/install.html
* A DigitalOcean account and an API key https://www.digitalocean.com/?refcode=2c62404bb57

Once you have installed Terraform and Packer, the first step is to create the Images for the Salt Master and a Salt Minion using Packer.

All of the code used in this post is available on Github: https://github.com/gordonmurray/terraform_salt_digitalocean

Create a file called packer/variables.json with the following code, replace the ‘xxx’ string with your DigitalOcean API key. This will allow Packer to store the Images in your DigitalOcean account

{
"do_token": "xxx"
}

Next create a file called packer/salt_master_do_image.json with the following content:

{
  "variables": {
    "do_token": ""
  },
  "builders": [
    {
      "droplet_name": "salt",
      "snapshot_name": "salt",
      "type": "digitalocean",
      "ssh_username": "root",
      "api_token": "{{ user `do_token` }}",
      "image": "ubuntu-18-04-x64",
      "region": "lon1",
      "size": "512mb"
    }
  ],
  "provisioners": [
    {
      "type": "shell",
      "scripts": [
        "scripts/install_salt_master.sh"
      ]
    }
  ]
}

The ‘builders’ section allows you to make choices such as Image name, location, server size and type. In this example I am using Ubuntu version 18.04 and a small 512mb size Droplet.

The ‘provisioners’ section allows you to call additional scripts to install and run any commands to customize the Image further. In this example I am using a simple Bash script to install the Salt master and Salt minion. You could also use Ansible here if you wanted to.

For the Bash script to install necessary items, create a new file called packer/scripts/install_salt_master.sh with the following lines:

#!/usr/bin/env bash
set -xe
sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get install salt-master -y
sudo service salt-master status

These steps will perform an apt update on the server and install salt-master.

Next create a file called packer/salt_minion_do_image.json with the following content:

{
  "variables": {
    "do_token": ""
  },
  "builders": [
    {
      "droplet_name": "salt-minion",
      "snapshot_name": "salt-minion",
      "type": "digitalocean",
      "ssh_username": "root",
      "api_token": "{{ user `do_token` }}",
      "image": "ubuntu-18-04-x64",
      "region": "lon1",
      "size": "512mb"
    }
  ],
  "provisioners": [
    {
      "type": "shell",
      "scripts": [
        "scripts/install_salt_minion.sh"
      ]
    }
  ]
}

Create a new file called packer/scripts/install_salt_minion.sh with the following lines:

#!/usr/bin/env bash
set -xe
sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get install salt-minion -y
sudo service salt-minion status

You now have 2 Packer files which you can use to build the Salt Master and Minion Images and 2 Bash scripts to install the Salt Master and Salt Minion files.

You can validate the Packer files using :

packer validate -var-file=variables.json salt_master_do_image.json
packer validate -var-file=variables.json salt_minion_do_image.json

You can build the Images using :

packer build -var-file=variables.json salt_master_do_image.json
packer build -var-file=variables.json salt_minion_do_image.json

Once both builds have finished, in the Images section of your DigitalOcean account, you should see two new Images, ready to be used.

digitalocean imagesThose Images won’t cost you anything to store. You’ll be able to use them any time in the future as templates for any new Droplets you create. Its also possible to change their owner, so you can send Images to other DigitalOcean accounts.

Now that the Images are ready, we’ll use Terraform to create 4 new Droplets, 1 Salt Master and 3 Salt Minions.

Just like Packer, we need to give Terraform access to your DigitalOcean account, create a new file called variables.tfvars with the following content, changing the ‘xxx’ string to your DigitalOcean API key.

do_token = "xxx"

Next create main.tf:

variable "do_token" {}

# Configure the DigitalOcean Provider
provider "digitalocean" {
    token = "${var.do_token}"
}

This will tell Terraform to get the DigitalOcean Provider and use your API key.

Next create ssh_key.tf, this will tell Terraform to create an SSH key in your Digital Ocean account which can be used to SSH in to the Droplets later if needed.

module "my_ssh_key" {
  source     = "./modules/digitalocean/ssh_key"
  name       = "my_ssh_key"
  public_key = "${file("/home/ubuntu/.ssh/id_rsa.pub")}"
}

Update the path in ‘public_key’ to point to your own public key file.

Next create droplet.tf, this is where Terraform will use the Images we have created and create new Droplets.

# get the pre made salt master image
data "digitalocean_image" "salt" {
  name = "salt"
}

# get the pre made salt minion image
data "digitalocean_image" "salt-minion" {
  name = "salt-minion"
}

# generate a short random string
resource "random_string" "first" {
  length  = 8
  special = false
  upper   = false
}

# generate a short random string
resource "random_string" "second" {
  length  = 8
  special = false
  upper   = false
}

# generate a short random string
resource "random_string" "third" {
  length  = 8
  special = false
  upper   = false
}

# create salt master
module "salt" {
  source             = "./modules/digitalocean/droplet"
  image              = "${data.digitalocean_image.salt.image}"
  name               = "salt"
  region             = "lon1"
  size               = "512mb"
  backups            = "false"
  monitoring         = "true"
  ssh_keys           = ["${module.my_ssh_key.ssh_fingerprint}"]
  private_networking = "true"

  content     = "#interface: 0.0.0.0"
  destination = "/etc/salt/master"

  remote_exec_command = "salt-key -A -y"
}

# create salt minion with a unique hostname
module "salt-minion-1" {
  source             = "./modules/digitalocean/droplet"
  image              = "${data.digitalocean_image.salt-minion.image}"
  name               = "salt-minion-${random_string.first.result}"
  region             = "lon1"
  size               = "512mb"
  backups            = "false"
  monitoring         = "true"
  ssh_keys           = ["${module.my_ssh_key.ssh_fingerprint}"]
  private_networking = "true"

  # add the master IP address and the minions host name to a salt config file
  content     = "master: ${module.salt.droplet_ipv4_private}\nid: salt-minion-${random_string.first.result}"
  destination = "/etc/salt/minion"

  # restart salt-minion on the host to take in the config file change
  remote_exec_command = "sudo service salt-minion restart"
}

# create salt minion with a unique hostname
module "salt-minion-2" {
  source             = "./modules/digitalocean/droplet"
  image              = "${data.digitalocean_image.salt-minion.image}"
  name               = "salt-minion-${random_string.second.result}"
  region             = "lon1"
  size               = "512mb"
  backups            = "false"
  monitoring         = "true"
  ssh_keys           = ["${module.my_ssh_key.ssh_fingerprint}"]
  private_networking = "true"

  # add the master IP address and the minions host name to a salt config file
  content     = "master: ${module.salt.droplet_ipv4_private}\nid: salt-minion-${random_string.second.result}"
  destination = "/etc/salt/minion"

  # restart salt-minion on the host to take in the config file change
  remote_exec_command = "sudo service salt-minion restart"
}

# create salt minion with a unique hostname
module "salt-minion-3" {
  source             = "./modules/digitalocean/droplet"
  image              = "${data.digitalocean_image.salt-minion.image}"
  name               = "salt-minion-${random_string.third.result}"
  region             = "lon1"
  size               = "512mb"
  backups            = "false"
  monitoring         = "true"
  ssh_keys           = ["${module.my_ssh_key.ssh_fingerprint}"]
  private_networking = "true"

  # add the master IP address and the minions host name to a salt config file
  content     = "master: ${module.salt.droplet_ipv4_private}\nid: salt-minion-${random_string.third.result}"
  destination = "/etc/salt/minion"

  # restart salt-minion on the host to take in the config file change
  remote_exec_command = "sudo service salt-minion restart"
}

This file may look like there is a lot going on when you first look at it, though the the sections are straight forward. The ‘data’ section at the beginning of the file is getting the pre-made Images we made earlier in to variable names we can reference later in the file.

The ‘resource’ sections are creating random strings, so that we can give the Salt Minions unique host names, otherwise they’ll have the same host name when we create them.

The ‘modules’ are where the real work is happening. Each module section is a Droplet that Terraform will create. There is 1 for a Salt master and 3 for Salt Minions. The module is describing the Droplet that will be made, including information such as the Image, the host name, region, public key for access and the size of the Droplet to use.

Two lines worth highlighting though are :

 # add the master IP address and the minions host name to a salt config file
 content = "master: ${module.salt.droplet_ipv4_private}\nid: salt-minion-${random_string.first.result}"
 destination = "/etc/salt/minion"

Those lines are applied to the Minions only and they are used to create a file called /etc/salt/minion within each Minion. This file will contains both the Private IP address of the Salt master Droplet so they know what to connect to, and also the unique hostname of the Minion itself, otherwise each Minion will contact the Master under the host name of ‘salt-minion’

All 4 Modules also contain a ‘source’ field and each one points to a single module in the modules folder called modules/digitalocean/droplet/main.tf. Creating Terraform Modules allows us to create re-usable blocks of code we can use in other Terraform projects.

resource "digitalocean_droplet" "default" {
  image              = "${var.image}"
  name               = "${var.name}"
  region             = "${var.region}"
  size               = "${var.size}"
  backups            = "${var.backups}"
  monitoring         = "${var.monitoring}"
  ssh_keys           = ["${var.ssh_keys}"]
  private_networking = "${var.private_networking}"

  provisioner "file" {
    content     = "${var.content}"
    destination = "${var.destination}"
  }

  provisioner "remote-exec" {
    inline = [
      "${var.remote_exec_command}",
    ]
  }
}

# Send outputs from this resource back out
output "droplet_ipv4" {
  value = "${digitalocean_droplet.default.ipv4_address}"
}

# Send outputs from this resource back out
output "droplet_ipv4_private" {
  value = "${digitalocean_droplet.default.ipv4_address_private}"
}

We need to create 1 more filemodules/digitalocean/ssh_key/main.tf, a Module for the Key pair we used earlier:

resource "digitalocean_ssh_key" "default" {
  name       = "${var.name}"
  public_key = "${var.public_key}"
}

# Send outputs from this resource back out
output "ssh_fingerprint" {
  value = "${digitalocean_ssh_key.default.fingerprint}"
}

Our next step is to initialise Terraform and run its ‘plan’ command to see what Terraform will create:

terraform init
terraform plan

This command will show you and output consisting of 5 items to create, the public key, the Salt Master and 3 salt minions.

To go ahead and create them, run:

terraform apply

This will show you the same output as the ‘plan’ command again with a N/Y Prompt for it to go ahead and create the Droplets, you can press Y to continue or you can use the following to skip the confirmation and continue to build the Droplets

terraform apply -auto-approve

In a short few minutes, you will now have a Salt Master and 3 Salt Minions running ready to you.

To determine the IP addresses of the new Droplets, you can look at your DigitalOcean Droplets page, or you can type:

terraform show

This will show you the IP adresses and other information about the Droplets Terraform has created.

To log in to the Salt Master, you can use:

ssh root@{ip address of salt master}

As you are using a Public key, you will be able to log directly in.

You can issue the following command to see the Salt Minions that are tring to connect to the Master:

salt-key -L

You can accept all Minions using:

salt-key -A -y

Your Salt Master can no issue commands to each of the Minions, for example:

salt * cmd.run 'hostname -a'

You will get back a response from each Salt Minion with their unique host name.

You now have a small, functioning and affordable Salt cluster to continue to use to learn about Salt, grains, pillars and Services.

All of the code used in this post is available on Github: https://github.com/gordonmurray/terraform_salt_digitalocean

My notes from “Practical Monitoring” by Mike Julian

I finished this book earlier today, I enjoyed it.

I wanted to write up some notes I took along the way, not as a book review or anything, just to try and help me to remember some of the lessons learned.

  • The message that monitoring is not just for sysadmin/ops engineers is mentioned a couple of times in the book.
  • Focus on monitoring what is working, that is, what makes the app work, instead of a broader metrics like CPU / Memory
  • Focus on the over all monitoring mission, and not just to the specific tool(s) in use at the moment
  • Components of a monitoring service are: Data collection, Data storage, Visualization, Analytics and reporting, Alerting.
  • Use a SAAS tool for monitoring, it costs more than you think to develop in house. Unless you’re Google or Netflix don’t do it.
  • For alerting, automate the solution and remove the alert of possible. If human action is needed, use run books to list out the options and steps to resolve.
  • Incident response management guidelines
  • Front end monitoring is often overlooked, despite having an impact on revenue, page load time can increase over time and impacts on users happiness.
    • webpagetests.org is worth looking at in this area. For example it’s possible to measure front end performance impact of every pull request. ( look up webpagetest private instances )
    • For APM, statsD may be worth a look. Node based.
  • Monitoring deployments is often overlooked but worth doing to help correlate deployments against increased error rates in an API for example.
  • Kook in to distributed tracing. Ideal for micro service architectures.
  • Good info on use of /proc/meminfo on page 94, related to server monitoring and how to read it’s output correctly, as well as grepping syslog for OOMKiller, meaning the system is looking to free up some memory.
  • iostat good for disk stats, specially to see transfers per second (tps) also called IOPS.
  • Stop using SNMP. Insecure. Hard to extend. Opt for push based such ad collectd or telegraf
  • For databases, keep an eye in slow queries and IOPs.
  • For queues, such as RabbitMQ, start by monitoring queue size and messages per second
  • For Caches, such as Redis aim for 100% hits. Not always possible to do, but worthwhile aiming for.
  • Auditd is useful for monitoring user actions on a server. It can be told to monitor specific files too. Ideal for watching config files for changes. Use audisp-remote to send logs.
  • Security monitoring
    • Look in to Cloud Passage and Threat Stack
    • Use rkhunter and use a cron entry to keep it updated daily. Set up alerts for warnings in its logs.
    • Look in to Network Intrusion Detection (NIDs) and network taps to analyse traffic for stuff that has gotten passed the firewall

Storing files to multiple AWS S3 buckets

Option 1, Single bucket replication (files added to Bucket A are automatically added to Bucket B)

If you aim to store files to a second S3 bucket automatically upon uploading, the built in “Cross Region Replication” is the method to use.

Its very easy to set up with just a few clicks in the AWS console.

  • Select the properties of the S3 bucket you want to copy from and in “Versioning”, click on “Enable Versioning”.
  • In “Cross Region Replication”, click on “Enable Cross-Region Replication”
  • Source: You can tell S3 to use the entire bucket as a source, or only files with a prefix.
  • Destination Region: You need to pick another region to copy to, it doesn’t work in the same region
  • Destination Bucket: If you have created a bucket in that region already you can select it, or create a bucket on the fly.
  • Destination storage class: You can chose how files are stored in the destination bucket.
  • Create/select IAM role: This will allow you to use an existing IAM role or create a new role with the appropriate permissions to copy files to the destination bucket.

Once you press Save, Cross Region Replication is now set up. Any files you upload to the source bucket from now on will automatically be added to the destination bucket a moment later.

It doesn’t copy across any pre-existing files from the source, only new files are acted upon.

Also, Cross Region Replication can’t (currently) be chained to copy from a source to more than one destination, however there’s a way to do that using Lambda.

Option 2, Multiple bucket replication (files added to Bucket A are automatically added to Bucket B,C,D )

In a nutshell, AWS Lambda can be used to trigger some code to run based on events, such as a file landing in an S3 bucket.

The following steps help create a Lambda function to monitor a source bucket and then copy any files that are created to 1 or more target buckets.

Pre-Lambda steps

  1. Create 1 or more Buckets that you want to use as destinations
  2. Clone this node.js repository locally (https://github.com/eleven41/aws-lambda-copy-s3-objects) and run the following to install dependencies
    1. npm install async
    2. npm install aws-sdk
    3. compress all the files in to a Zip file
  3. In AWS’s ‘Identity and Access management’, click on ‘Policies’ and click ‘Create Policy’, copy the JSON from the ‘IAM Role’ section of the above repository : https://github.com/eleven41/aws-lambda-copy-s3-objects
  4. In IAM Roles, create a new Role. In ‘AWS Service Roles’, click on Lambda and select the Policy you created in the previous step.

Lambda steps

  • Log in to the AWS Console and select Lambda, click on “Create a Lambda function”.
  • Skip the pre made blueprints by pressing on Skip at the bottom of the page.
  • Name: Give your lambda function a name
  • Description: Give it a brief description
  • Runtime: Choose ‘Node.js’
  • Code entry type: Choose ‘upload a .zip file’ and upload the pre-made Zip file from earlier, no changes are needed to its code
  • Handler: Select ‘index.handler’
  • Role: Select the IAM role created earlier from the Pre-Lambda steps.

You can leave the remaining Advanced steps at their default values and Create the Lambda function.

  • Once the function is created, it will open its Properties. Click on the ‘Event Sources’ tab.
    • Event Source Type: S3
    • Bucket: Choose your source bucket here
    • Event Type: Object Created
    • Prefix: Leave blank
    • Suffix: Leave blank
    • Leave ‘Enable Now’ selected and press ‘Submit’
  • Go back to your original source S3 bucket, create a new Tag called ‘TargetBucket’
  • In the ‘TargetBucket’ value add a list of target buckets, separated by a space, that you want files copied to. If the buckets are in different regions you’ll need to specify, for example:
    • destination1 destination2@us-west-2 destination3@us-east-1

You can use Lambda’s built in Test section to test the function works well. Don’t forget to change the Test script to specify your source bucket.

If there are errors, there will be a link to Cloud Watch error logs to diagnose the problem.

Any files added to the source bucket will now automatically be added to the one or more target buckets specified in the ‘TargetBucket’ value of the original bucket.