Automatically Backup your EC2 Instances using AWS Backups & Terraform

Overview

For a while, taking automated EC2 backups in AWS either involved using the Data Lifecycle Manager (DLM) or resorting to some CloudWatch trickery (using Events). Both had their pros and cons and got the job done, but neither felt like a full-fledged solution designed solely for backups. AWS probably realized this and released AWS Backups last year (2019).

In this article we’ll create a completely automated backup solution using AWS Backups and Terraform.

This article was written as part of Akatsuki’s 2020 Advent Calendar

Rather than boring you with any more overviews or other long explanations, let’s jump right into it shall we!

What we’ll do

• We’ll automatically create scheduled Image backups of an EC2 instance
• We’ll also set a lifecycle policy on the backups so they’re automatically deleted after a 1 week retention
• We’ll create the entire infrastructure (including a target EC2 instance to backup) in Terraform so that it’s easy to maintain in future

Prerequisites

Before we start here are some prerequisites for those following along

Knowledge

• Basic knowledge of Terraform
• Basic working knowledge of AWS (or another cloud provider)
• And as always, an open mind :)

Tools

• Terraform
• An AWS account and AWS CLI
• Your favourite IDE or a Text Editor

I’ll be using a Mac for this article but it should be fairly easy for Windows users to adapt it to their environment

Note you’ll be charged for any resources you create! So please keep that in mind if you use terraform to create the infrastructure!

If you want to skip all the explanation and just see the code, the complete project can be found on GitHub

To keep things simple we’ll be using a local state in this article but the GitHub project contains a backend.tf file that can be uncommented and modified to save your state to S3 and to enable state locking

Ok let’s start! :)

Setup

You can skip this section if you already have the Terraform and the AWS CLI installed and working

Terraform

Install Terraform first on your machine (if you don’t have it installed already)

I’ll be using v0.13.3 but the latest one should be fine. Verify it’s running

$ terraform --version
Terraform v0.13.3

AWS CLI

Install the latest AWS CLI and configure your credentials

$ aws configure

You’ll be asked for your AWS Access Key ID and Secret Access Key, which you’ll find here

Once completed your credentials file will be created at ~/.aws/credentials (on Mac and Linux) or %UserProfile%\.aws\credentials (on Windows)

Make sure your credentials are configured correctly by running a test command using the AWS CLI which should give you your account details

$ aws sts get-caller-identity
{
    "Account": "123456789012", 
    "UserId": "AR#####:#####", 
    "Arn": "arn:aws:sts::123456789012:assumed-role/role-name/role-session-name"
}

If you get an error at this point, your credentials are not configured correctly so I’d recommend checking out the AWS documentation for more information

Instance

Create a directory to store all your terraform files (I’ll refer to this as the project directory) and create a versions.tf file and add the terraform and provider version constraints

terraform {
  required_version = "~> 0.13.0"

  required_providers {
    aws = "~> 3.0"
  }
}

Next let’s initialise Terraform inside the project directory.

$ cd ~/examples/aws-backups-terraform
$ terraform init
Initializing the backend...
Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 3.0"...
- Installing hashicorp/aws v3.21.0...
- Installed hashicorp/aws v3.21.0 (signed by HashiCorp)
Terraform has been successfully initialized!

Now let’s create an EC2 instance configuration that we’ll use as the target for the backups

Create an instances.tf file in the project directory with the following code

You’ll notice that I’m adding a bunch of tags to the instance to make it easier to identify later and in case you want to create a Resource Group later

I’m also adding a special tag Backup=”true”, we’ll be using this when targeting this instance for backups

You’ll also notice a bunch of variables defined in the file. Let’s define values for these next. Create a terraform.tfvars.json file. Make sure to replace the value of the public_key variable with your public key. You’ll be using it to create the AWS Keypair to attach to the EC2 instance.

Also feel free to change the other variable values like the region as needed. Note for security reasons I’m not attaching a public IP to the instance, but if you plan to log into the instance you can set the attach_public_ip variable to true. Or alternatively you can use EC2 Session Manager to do access the instance more securely

{
  "project": "backups-example",
  "region": "us-east-1",
  "profile": "default",
  "public_key": "<YOUR_PUBLIC_KEY>",
  "attach_public_ip": false
}

Let’s check for errors

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create
Terraform will perform the following actions:
# aws_instance.example-server will be created
  + resource "aws_instance" "example-server" {
      + ami                          = "ami-0be3f0371736d5394"
...
...
Plan: 2 to add, 0 to change, 0 to destroy

Terraform should report 2 new resources to add — The EC2 instance and it’s corresponding keypair

Let’s apply this and proceed with the rest of the resources. If you‘re following along you can enter yes when prompted to create the resources. Otherwise you can skip this and all subsequent terraform apply steps

$ terraform apply
An execution plan has been generated and is shown below
...
...
Plan: 2 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.
Enter a value:

IAM Roles and Policies

Before we create the Backup resources, we’ll have to create the necessary IAM roles and policies for it to reference. Having all the necessary policies for the backup to work is probably the most troublesome part, but luckily you can use this as a reference :)

Create an iam.tf file in the project directory

I’m attaching a bunch of policies that are required by the AWS Backups service for taking backups and restoring from a backup. Note that restoring from a backup also needs an additional PassRole policy to be attached

Run terraform plan followed by terraform apply to create the roles and policy attachments

Backup Resources

Finally, let’s create the main part of this infrastructure — the backup resource

Create a new backups.tf file

There are several things going on here

• We’re first creating a Backup Vault — this will store all our backups and allows you to search, restore, delete or copy them to a different region. It’s basically a management dashboard for your backups
• Next we create a Backup Plan — this is where we specify the schedule, retention time, target vault and some other metadata for the backups. It’s a good idea to tag your backups so they’re easy to filter later on. The backup schedule is specified in cron format and the backups are automatically deleted once the retention period passes. Note however, that there can be a slight delay after the retention period has been exceeded for the deletion to occur and it doesn’t happen
• Finally we create a Backup Selection — this is where you select the resources you want to backup and associate the previous Backup Plan to them. We’re targeting all resources with the Backup tag set to true. Note that the type of Backup changes based on the target resource. Since we’re targeting an EC2 instance an AMI Backup will be created. If we were to target a disk instead, a Snapshot would be created

Let’s do a final terraform plan followed by terraform apply to create the backup resources!

With this you’re all set!

Verify the Backups and Cleanup

You can verify your backups in the AWS Console under the Backup vaults section. Your backups should automatically be triggered based on your schedule and appear here

You can also view all your Backup and Restore Jobs under the Jobs section. This is useful for referencing your backup history including any Failed Jobs

Finally don’t forget to cleanup your resources to reduce your bill!

$ terraform destroy

Summary

In this article we created an automated backup solution using Terraform. The backups were created on a schedule and were automatically deleted after a specified retention period

The source for this project can be found on GitHub. We used a local state in this article but the project contains a backend.tf file that can be uncommented to save your state to S3 and to enable state locking. Just make sure you modify the attributes before using it!

Thanks and see you again on the next adventure :)