Updating Windows and Linux servers

Patch management is critical to keep your systems up to date. To ensure ongoing performance and to avoid issues such as unpatched vulnerabilities. In this post, you will gain practical experience in patching operating systems such as Windows Server and Linux based machines in both a manual and automated way using tools such as Ansible.

We will cover the following topics:

  • Patching Windows Servers
  • Patching Windows Servers using Ansible
  • Patching Linux servers
  • Patching Linux servers using Ansible
  • Replacing servers with pre-patched servers
  • Performing updates to AWS Relational Database Service (RDS) instances

Patching Windows Servers

Microsoft windows provides a service called Windows Update which is designed to keep your operating system up to date with the latest bug fixes or security patches supplied by Microsoft.

Windows Update is usually enabled by default on a Windows PC. It is important to enable this service and allow it to keep your system up to date. In our next section we will make sure Windows Update is enabled and functioning.

Enable Windows Update to install updates automatically

From your windows desktop, use the following steps to open Windows Update and ensure it is set up to install Windows Updates automatically:

  1. Press on Start | select Control Panel
  2. Once the Control Panel window has opened, click on System and Security
  3. Once the System and Security window has opened click on Windows Update
  4. Once the Windows Update window has opened, click on Change settings to the left of the screen:
  5. In the Important updates make sure that Install updates automatically (recommended) is selected.
  6. Using the Install new updates drop down menu, you can choose when to install new updates. Select a frequency and time that works for you, such as Every day at 06:00.
  7. Using the Recommended updates check box, you can choose to allow Windows to also install recommended updates along with important updates.
  8. Using the Who can install updates, you can choose to allow any users of your windows machine to install updates.
  9. Once you have made your choices, press on OK to apply.

With these changes in place, Windows will check for updates and keep itself up to date.

In our next section we will look at how to install Windows Updates manually.

Installing Windows Updates manually

There may be times when you would not like to enable automatic Windows updates and so you should apply any Windows updates manually.

  1. Press on Start | select Control Panel
  2. Once the Control Panel window has opened, click on System and Security
  3. Once the System and Security window has opened click on Windows Update
  4. Press on Check for updates on the left. Windows will spend some time searching for any updates that need to be applied as shown in the following screenshot:
  5. If you would like to go ahead, press on Install Updates. Windows update will begin downloading and installing one or more updates. The Window might close as it performs the updates in the background or it may stay open showing current progress.

Windows update installing an update In this section, we have covered a number of steps to ensure that the Windows Update service is running and is installing any necessary updates. In our next section, we will look at how to handle failed windows updates which may occur. Handling Failed Windows updates

When installing Windows updates, unfortunately some updates can fail to apply. Sometimes pressing on Try again can work in case there was a temporary connectivity issue that interrupted the download of a Windows Update.

If the problem persists, there is a Get help with this error link which can open a Help page that may help you do see the problem and fix it.

Another option when there is a failed update is to click on View update history to open a list of all updates applied as shown in the following screenshot:

In the list, look for updates with a Status of Failed. Take a note of the value starting with KB which stands for Knowledge Base and perform a search online for the value, such as KB915597. There will often be a result from a Microsoft Community forum at https://answers.microsoft.com

In this section, we looked at ensure that the Windows Update service is running well and also how to handle any failed Windows updates. This is a manual process which won’t work well if you have many servers. In our next section we will look at patching Widows server using Ansible to help automate the process.

Patching Windows Servers using Ansible

We will need to perform a number of steps first to configure our Windows server so that Ansible can connect and perform any tasks. Our first step is to make sure that remote connections are allowed to connect to the Windows server.

Allow Remote Desktop

On the Windows Server, click on Run and search for View Advanced System Settings and click the search result to open it.

Navigate to the tab called Remote and in the Remote Desktop section, click on Allow connections from computers running any version of Remote Desktop and press on OK.

Next, we will need to make sure that PowerShell, used by Ansible is up to date and that Windows is set up to allow Ansible to connect.

The following steps are taken from the Ansible documentation available at: https://docs.ansible.com/ansible/latest/user_guide/windows_setup.html. Also, Ansible requires that Powershell version 3.0 at a minimum is running on the target Windows machine.

Update Powershell

On the Windows machine, open up Windows PowerShell ISE as Administrator. Paste in the following content. Include the username and password that you usually use to log in to Windows and press on Run:

$url = “https://raw.githubusercontent.com/jborean93/ansible-windows/master/scripts/Upgrade-PowerShell.ps1”
$file = “$env:temp\Upgrade-PowerShell.ps1”
$username = “Administrator”
$password = “Password”
(New-Object -TypeName System.Net.WebClient).DownloadFile($url, $file)
Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Force
# Version can be 3.0, 4.0 or 5.1
&$file -Version 5.1 -Username $username -Password $password –Verbose

This PowerShell script will update PowerShell and change the Execution Policy in Windows to allow Ansible to operate. Depending on your system, this process may take a few minutes to complete.

Also, a warning that your Windows server will reboot automatically once it has upgraded Powershell so it is best to perform this step at a time you are comfortable for your Windows server to reboot.

Configure Windows for Ansible

The next step is to configure the Windows server for ansible

Download the following Powershell script from Ansible’s Github account and save it as locally as ConfigureRemotingForAnsible.ps1.

This script checks the current WinRM (PS Remoting) configuration and makes the necessary changes to allow Ansible to connect, authenticate and execute PowerShell commands. https://raw.githubusercontent.com/ansible/ansible/devel/examples/scripts/ConfigureRemotingForAnsible.ps1

You will be asked if you trust this file and if you would like to prevent it running. Press r and Enter to allow the script to run once.

If your Windows Server is hosted with AWS, you will need to open Port 5986 for HTTPS-WinRM in the Security Group used by your Windows server.

To test that Ansible can connect to the Windows host, run the following from your terminal:

ansible windows -i hosts -m win_ping

You should get back a result showing a Success message as shown in the following screenshot:

Once Windows is configured, we can begin to create our Ansible playbook.

Create a new file called windows.yml with the following content for our Playbook:


– hosts: windows
gather_facts: no
roles: – windows-update

This is our main playbook file and it will call a role called windows-update.
Next, we will create our main tasks file. Create a new file called roles/windows-update/tasks/main.yml with the following content:


– name: Search and download Windows updates without installing them
win_updates:
state: searched

This task will tell Windows to search for Updates, but not install any yet.

The final step is to create a file called Hosts, this will contain a list of one or more IP addresses of the Windows servers that you would like to update:

Hosts

[local]
localhost
[windows]
192.168.1.100

You will need to change the IP address in the hosts file to match the IP address of your Windows server.

If you are running Ansible from your own Windows desktop, you can omit the windows section and update your playbook to use hosts: local instead of hosts:windows to run the playbook locally on your machine.

Run the playbook using the following command:

Ansible-playbook windows.yml -i hosts

Next, let’s update the playbook to get Windows to install Security and Critical updates only. Edit the main.yml file and append the following:

– name: Install all security, critical, and rollup updates
Win_updates:
Category_names:
– SecurityUpdates
– CriticalUpdates
Reboot: yes

The Reboot: yes option will reboot the windows server if it is necessary to complete any Windows updates.

Run the playbook again using the following command:

Ansible-playbook windows.yml -i hosts

Congratulations! You have successfully written an Ansible playbook to keep one or more Windows servers up to date.

So far we have covered performing Windows updates both via manual steps and also in an automated way using Ansible. In our next section we will change to Linux servers and see how we also perform updates both manually and via Ansible.

Patching Linux servers

In this section we will look at some of the command line tools we can use to keep our Ubuntu based servers or desktops up to date. We will use Ubuntu’s package manager tool called Advanced Packaging Tool (APT) to perform the updates needed.

In your terminal use the following command. It will not make any changes to our system, it will only download package information from all configured sources:

apt update

This will download package information from sources in /etc/apt/sources.list but it will not perform any actual updates.

Next use the following command in your terminal to upgrade existing packages:

apt upgrade

Upgrade will read the updated package information and it will perform the updates to bring your packages up to date. It will provide a list of packages it will upgrade for you to approve before upgrading.

The next command in our terminal will update any dependencies:

apt dist-upgrade

A dist-upgrade is similar to an upgrade, however it can also handle any dependencies such as downloading additional items that an upgrade requires and may also remove older dependencies.

Now that we have performed our updates, we can perform some clean up steps. Use the following command in the terminal to clean up references to unnecessary packages

apt autoclean

autoclean will remove the local cache of retrieved package files that are no longer in use.

Use the following command in your terminal to fully remove any unnecessary packages:

apt autoremove

autoremove will remove any packages that were installed automatically as part of a dist-upgrade that are no longer needed.

We can put the above commands in to one script. Create a new file called update.sh with the following commands:

apt update
apt upgrade -y
apt dist-upgrade -y
apt autoclean -y
apt autoremove -y

The -y parameter will allow the command to continue without waiting for your user input.

While the above commands will ensure your linux based system is up to date with the latest packages, it is often not a good idea to run all of those commands on a live production server. You may have software on your live server that will no longer run well if packages are upgraded. A good example of this is if you have a web server using PHP and you upgrade the PHP version to the latest version, your application may stop working.

While it is possible to reverse some changes after performing upgrades, a better approach is to test that your application works on an upgraded system first. Later in this post we will look at approaches to handle this.

To run our update.sh script, run the following command:

sudo bash update.sh

This script will run all of the commands to update, upgrade and clean our locally installed packages.

When using an Ubuntu based system, there is an additional command that can be run to see if there is an update to the operating system you are running currently. If you are running a system with Ubuntu 18.04 LTS installed, the following command will show you that Ubuntu 20.04 LTS is available and will perform the upgrade for you if it is possible to do so based on the packages you have installed.

sudo do-release-upgrade

This command will upgrade Ubuntu to the latest version of the operating system that is available. A configuration file can be changed if you would like to stop release upgrades, or if you’d like to only upgrade to Long Term Support (LTS) versions of Ubuntu, or use a normal release upgrade to get the latest available version of Ubuntu.

To change the configuration file, you can change the ‘Prompt’ value in the file called /etc/update-manager/release-upgrades to ‘normal’, ‘lts’ or ‘never’ to suit your own preferences. ‘Lts’ is usually a good option to get the latest long term supported version.

Now that we know how to update a single Ubuntu server, in our next section we will use Ansible to perform the same overall steps in an automated process that could be used to update many servers automatically.

Patching Linux servers using Ansible

In this section, we will use Ansible to perform the same steps to update a linux server as we performed earlier to bring a server up to date.

We will use the folder structure of an Ansible Role, so that the finished Role can be used directly or easily added to our existing Ansible project from an earlier post in the ‘Using Ansible’ section.

First we will create the main file in our Ansible Role. It will instruct Ansible to run the role called update on the hosts in the all group and to connect using the user called ubuntu.

In your editor, create a new file called 03_update.yml with the following content:


– hosts: all
gather_facts: no
become: yes
user: ubuntu
roles:
– update

The field become: yes is applied as we will need Sudo permissions to perform the APT updates.

The gather_facts: no is applied as we do not need to gather information from the target host for this role and it allows the Playbook to run a little faster.

Next we will create the main file that will be used by our Ansible role. In your editor, create a new file called /roles/update/tasks/main.yml with the following content:

– name: Run apt update
apt:
update_cache: yes

– name: Run apt upgrade
apt:
name: “*”
state: latest

– name: Run apt dist-upgrade
apt:
upgrade: dist

– name: Run apt autoclean
apt:
autoclean: yes

– name: Run apt autoremove
apt:
autoremove: yes

This is our main tasks file where we will perform the same APT update steps as we have performed manually earlier in this post. We do not need any other files or templates for updating the target server, so the two files we have just created are enough to continue.

If you have kept your webserver creating using Ansible running from ‘Using Ansible’ then you can use the following command to run your playbook to update your webserver:

Ansible-playbook 03_update.yml –ask-vault-pass

Alternatively, instead of creating the new 03_update.yml paybook, you could edit your existing webserver.yml file and add the update role to the existing roles so that the updates would be performed before installing apache or PHP, so your webserver.yml file would become:

Roles:
– update
– apache
– php

Congratulations! You have successfully written an Ansible playbook to keep one or more Ubuntu servers up to date.

Safely updating production servers

In some cases it can be quite damaging to perform updates to an active server in production use. If a server contains a number of different packages, one or more of them might be updated and reach a state where they are no longer compatible with each other, potentially causing your software to stop working.

One approach to safely update a production server is to use one or more tools to create a new server, provision it with the exact same software and perform any updates there first. Once confirmed, the new server can replace the old server by reassigning its IP address or DNS values so it becomes the new production server. An earlier post includes a step by step guide on how to create a new server in this way in the section called Replacing a webserver using Packer, Ansible and Terraform.

Another approach that can be taken to avoid updating a server directly if you have not yet taken steps into automating your builds and deployments is to make a copy of an existing server. Once copied, perform the updates on the copy of the server first to make sure it is safe. Once you are sure the update process is safe, you could then apply the same update steps to the production server.

Replacing servers with patched servers earlier in this post we learned how to apply updates to an existing Windows or Linux server. We also learned that applying updates to a production server – that is, a server that is in active use may cause problems for users.

In this section we will use the AWS console and the AWS CLI to copy or clone an existing server to create an Amazon Machine Image (AMI).

This AMI can serve simply as a backup. This AMI can also be used as a template for testing any updates in isolation.

The AMI can also completely replace an existing server once it has been updated successfully.

Clone an EC2 instance using the AWS console

In the following steps we will use the AWS console to clone an existing EC2 instance to make an AMI available for testing or replacing.

  1. To begin creating an AMI from an existing EC2 instance, log in to your AWS Console and click on EC2 in the Compute section or by searching for EC2 in the Find Services search bar.
  2. Click on Running Instances to view any existing EC2 instances. If you were following along with earlier posts and left your webserver running, you should be able to see at least one EC2 instance.
  3. Next to an existing EC2 instance, select the checkbox and then click on the Actions drop down menu and select on Image and then Create Image.
  4. In the window that opens, you can give your image a name such as webserver, a description and a checkbox to prevent the EC2 instance from rebooting. Fill in each field to give your AMI a suitable name and a relevant description and select No Reboot
  5. In the same window, a shot list of disk volumes that are attached to your EC2 instance will be visible. You can leave this list unchanged unless you want to make specific changes to the volumes such as changing the disk size ( Note that if you change the disk size here, it will only change the disk size of the new EC2 instance that you create, it will not change your existing EC2 instance).
  6. When you are ready, click on Create Image. This will start the process of creating a copy of your existing EC2 instance.
  7. You can click on the View pending image link to see your AMI being created, or if you have already closed the window you can click on AMIs under Images from the navigation mentioned on the left of the screen.

Your AMI will remain in a Pending state for a short time though this will vary depending on the size of your EC2 instance.

We have now created an AMI using the AWS Console using a manual process. In our next section, we will see how to perform the same steps using the AWS CLI as the CLI approach will allow us to automate this process if needed.

Clone an EC2 instance using the AWS CLI

The steps we have just taken have created an AMI of our existing EC2 instance. This AMI can be used in a number of ways. If our existing EC2 instance were to become damaged in some way, this AMI could be used to create a new copy of the EC2 instance. The AMI can be used to copy

our Webserver and make it available to another geographical region in your AWS account or even make your AMI available to another AWS account.

To perform the same action of creating an AMI using the AWS CLI, you could use the following cmmand:

aws ec2 create-image –instance-id {you-instance-id} –name webserver-cli –no-reboot –region us-east-1

If you are unsure of the instance ID of your existing EC2 instance, you can view it in the AWS Console or use the AWS CLI to describe the EC2 instance as follows:

aws ec2 describe-instances –region eu-west-1 –filter ‘Name=tag:Name,Values=webserver’

This will give you a detailed output of the EC2 instance. You can limit the information returned to just show the tag name and instanceId as follows:

aws ec2 describe-instances –query ‘Reservations[*].Instances[*].[Tags[0].Value , InstanceId]’ –region us-east-1 –filter ‘Name=tag:Name,Values=webserver’

Once you create an image, the output will show an AMI ID.

Now that we have created an AMI, in our next section we will learn how to turn this image in to a working EC2 instance.Create an EC2 instance from an AMI using the AWS console

Earlier in this post we created an AMI from an existing EC2 instance. In this section we will reverse the process to create a new working EC2 instance from our AMI. Once the new EC2 instance is running, we can perform the updates we performed earlier in a safe environment that will not impact on users.Use the following steps to create an EC2 instance from the AMI:

  1. To begin creating a new EC2 instance from an AMI, log in to your AWS Console and click on EC2 in the Compute section or by searching for EC2 in the Find Services search bar.
  2. From the navigation menu on the left, select Images | AMIs
  3. This section will list any AMIs already in your account. If you followed the steps from the previous section, you should have at least one AMI called webserver in the list.
  4. Select the Webserver AMI using the checkbox to the left of the AMI and click on Launch.
  5. You will be asked to select an Instance Type. Select t2.micro for a small low cost server if you are unsure of the type of server to use and click on Next: Configure Instance Details (It is tempting to press on Review and Launch, however this will make a number of assumptions and you may end up with a Webserver that you cannot connect to).
  6. In the Configure Instance section, select the VPC you would like to use from the Network drop down menu and press on Next: Add Storage.
  7. In the Add Storage section, you can leave the existing choices in place, unless you would like to give the new server some larger disk space and click on Next: Add tags
  8. In the Add Tags section add any option tags you would like to use here. For example, a Tag with a key of Name and a Value of Webserver – New. Once you are ready to continue press on Next: Configure Security Group.
  9. In the Security Group section, select on Select an existing security group and in the list look for the webserver-security-group we created earlier in the book when creating the webserver. Once you are ready to continue press on Review and Launch
  10. On this Review page you can view your choices and when you are ready to continue press on Launch
  11. On the modal window that pops up, select Choose and existing pair from the first drop down menu. From the second drop down menu, select ansible-webserver from the list which we created earlier. You can also create a new Key pair by choosing Create a new key pair if you wish. This will allow you to create a new key pair and download it. Make sure you have selected I acknowledge that I have access to the selected private key file and press on Launch Instances.
  12. The next screen will let you know that your instance is being created. Press on View Instances to go back to the main list of your EC2 instances and you should now see a new EC2 instance called Webserver – New if you filled in the optional Tags in the earlier step.
  13. Take a note of the IPv4 Public IP address of the new webserver so that you can connect to the instance

Once the Instance State shows Running, you can now connect to the instance using Remote Desktop Protocol (RDP) if it is a Windows based server or using SSH if it is a Linux based server.

Once you are connected, you can perform one of the actions above to perform the updates on your server using the manual or automated approach using Ansible.

When you have your new instance updated successfully, you will know if the updates are safe to apply to your original server in production. You could also start to use this new server instead

and update your domain name or DNS records to use this server instead and remove the old one.

In the same way se used the AWS Console and then the AWS CLI to create an AMI. In our next section, we will use the AWS CLI to create an EC2 instance using the AMI.

Create an EC2 instance from an AMI using the AWS CLI

Creating an EC2 instance from an AMI is a little different, you’ll need to know a couple of details first before you can create the instance.

The following API commands can be run in a single line, the line breaks represented by the backslash \ is to make the examples easier to read and will also work in the command line.

You will need to know the AMI ID and your existing security group ID before we can continue.

To use the AWS API to determine the ID of your AMI, use the following command:

aws ec2 describe-images \
–owners self \
–filters “Name=name,Values=webserver”

This command will return a JSON object with the full details of the Image we created earlier. You will be able to see the AMI ID within the JSON but if you’d like to get the AMI ID only, you can use the query parameter to reduce the information returned as follows:

aws ec2 describe-images \
–owners self \
–filters “Name=name,Values=webserver”
–query ‘Images[*].{Image:ImageId}’

In the output, you will see a short string in the format of ami-abc1234567890. The actual value you see will be slightly different but copy it for now and we will use it shortly.

To use the AWS API to determine the ID of your existing security group, use the following command:

aws ec2 describe-security-groups \
–group-names webserver-security-group \
–query ‘SecurityGroups[*].{Group:GroupId}’

Take note of the value that is returned, it will look similar to sg-1234567890.

Once you have your AMI and Security Group ID, add those details to the following command and run the completed command to start the new EC2 instance:

aws ec2 run-instances \
–image-id [add your AMI ID here] \
–instance-type t2.micro \
–key-name ansible-webserver \
–security-group-ids [add your security group ID here]\
–tag-specifications ‘ResourceType=instance,Tags=[{Key=Name,Value=Webserver-CLI}]’

In the output, take a note of the InstanceId value. We will use this value to get the IP address of the new instance so we can connect to it, it will look something like i-1234567890

Use the following command to retrieve the public IP address of the new instance:

aws ec2 describe-instances \
–instance-id {instance-id} \
–query ‘Reservations[*].Instances[*].NetworkInterfaces[*].Association.PublicIp’

You can now connect to the instance using RDP if it is a Windows based server or using SSH if it is a linux based server.

Once you are connected, you can perform one of the actions above to perform the updates on your server using the manual or automated approach using Ansible.

If you would like to remove your new EC2 instance after you have finishing using it, you can run the following command:

aws ec2 terminate-instances –instance-ids {instance-id}

In this section we used the AWS CLI to take a copy of an existing server in the form of an AMI. We used this AMI to create a new working server to use to test our updates. This can be a lengthy process, but it can help provide a safe place to test your updates before applying to production servers.

In our next section we will move away from working on EC2 instances and instead take a look at performing updates to AWS Relational Database Service (RDS) instances.

Performing updates to AWS Relational Database Service (RDS) instances

AWS RDS provides two different mechanisms to help you to keep your RDS instances up to date in relation to security or version updates.

The first is called Auto Minor version and is an optional parameter you can apply to your RDS instances. If you are using a particular version of mySQL such as version 5.7.21 and a new minor version such as 5.7.22 becomes available, then the RDS instance will automatically update to the next version during your defined Maintenance window.

The next mechanism is the Maintenance Window. This process allows you to choose a day of the week and a period of time within that day to allow Maintenance to occur automatically. This kind of maintenance can be the auto minor version we covered a moment ago, but it can also be a larger upgrade you have chosen to perform such as a mySQL version change from version 5.7.22 up to version 8.0 or increasing disk space, or changing the class of the server from an M5 to and R5. It is even possible to make several changes in one maintenance window.

The maintenance window allows you to control when these changes occur. It is recommended to perform changes such as this at an off-peak time when your database is not in use or is receiving minimal usage.

RDS also contains an ability called Multi Availability Zones (AZ) which is a process whereby RDS will create a copy of your RDS instance in a separate zone for high availability. This means if your primary RDS instance experiences some failure, the other instance can take over.

MultiAZ is not directly related to performing updates, however it can facilitate applying updates minimizing interruption for your database users. If you have MultiAZ enabled and you have chosen to perform a significant update to your RDS instance during a maintenance window, then RDS will apply the changes to the standby instance and once it is ready, RDS will ‘fail over’ to this updated instance, minimizing the time that your database users may be interrupted.

Summary

In this post, we learned a number of different approaches on how to perform updates to Windows or Linux based servers. We looked at how to perform those updates manually as well as perform the same steps in code using Ansible. We also looked at how to clone existing instances to use them as a platform for testing updates first or to become replacement instances instead. We also looked at the options available to us when performing updates to AWS Relational Database Service (RDS) instances.

In our next post, we will review a number of ways to monitor our infrastructure once it is running

Leave a Reply

Your email address will not be published. Required fields are marked *