Terraform 101

Terraform is an infrastructure as code automation tool that can be used with multiple cloud platforms through the use of a concept of distinct plugins called “providers” such as Azure, AWS and Google Cloud.

Terraform uses declarative domain specific language to describe a wide range of cloud resources and a desired state for the collection of resources under consideration.

This document describes how to install Terraform and begin working with the tool to create some basic infrastructure in Azure.

Installation

Terraform is available for Windows, MacOS, Linux, FreeBSD and Solaris. It can be downloaded from terraform.io/downloads.html. The process is straightforward; once downloaded, extract the tool and ensure that the Terraform folder is added to your systems PATH variable.

For example, on Windows, use the control panel, environment settings for your local account and add the folder in which you placed the Terraform binary. Perhaps this was C:\User\username\bin, depending on your preference. On Linux, simply add to your user shell .bashrc or preferred shell profile.

The binary is standalone, so nothing else is required, provided the path is correct, Terraform should now work for you:

$ terraform
Usage: terraform [global options] <subcommand> [args]

The available commands for execution are listed below.
The primary workflow commands are given first, followed by
less common or more advanced commands.

Getting Started

Terraform configurations consist of resources optionally arranged into modules. The resources are declarations, or descriptions of the infrastructure resources you wish to create and manage. The resources are declared using Terraform’s own configuration language called HCL. The core purpose of a module is to group related resources together, for example, you might group a virtual machine, network interfaces and disks together for a common application.

There is always a root module which is where the processing of the configuration starts. Further resources and child modules are created under the root module.

Every module will contain a group of files with the extension “.tf”. The required files are :

  • main.tf - Declares all the resources in this module
  • variables.tf - Contains input variables for the module
  • outputs.tf - Contains the outputs that the module should include

When a module is applied, perhaps from the command line using “terraform apply”, it will need input for all the declared variables. In the case where a large number of variables are required, a “variable file” can be used and passed to terraform with the “-var-file” argument:

$ terraform apply -var-file=”myvmvariables.tfvars”

Providers

Terraform is able to work with a wide range of hypervisors and cloud platforms. This is achieved via the use of separate “providers''. For example, there is an AWS and an Azure provider. Each provider contains the required logic to create the declared resources in the respective platform.

Declaring the provider in the HCL is required and is very straightforward.

provider "azurerm"  {  
}

Authentication

For Azure, it is possible to use Terraform from either of the two command line options that Azure offers - the Azure Powershell module or the Azure CLI. In this article, we will discuss the use of Azure CLI only.

First, simply ensure that you are logged into your Azure account using the Azure CLI.

> az account list --output table
Name           CloudName    SubscriptionId                        State    IsDefault
-------------  -----------  ------------------------------------  -------  -----------
Pay-As-You-Go  AzureCloud   22222222-bbbb-4444-aaaa-21111111119  Enabled  True

Manual login is a perfectly acceptable way to manage resources with Terraform if you are planning to run Terraform on demand at the command line. However, for automated calls or for placement within development and integration pipelines, a new method is required that doesn’t require the manual login process of entering credentials.

This is achieved by logging into Azure using a Service Principal and a Client Secret. From a logged-in AZ CLI shell, this can be accomplished as follows:

> az ad sp create-for-rbac --role=”Contributor” --scopes=”/subscriptions/[SUB-ID]” --name “Azure-terraform-role”

The output of that command, if successful, should give you the following items in output:

  • appID
  • displayName
  • name
  • Password
  • tenant

Record those details securely, they are your sensitive credentials for access to your Azure account for authenticating automation tasks. The credentials can then be tested at the command line with a test login.

> az login --service-principal -u <appID> -p <Password> --tenant <tenant>

If the service principal worked, the details can then be given to Terraform. In your local source repository, or the location in which you are build out your Terraform project structure, you can create the variables.tf file that will declare the placeholders for your credential variables:

# Azure Subscription Id
variable "azure-subscription-id" {
    type = string
    description = "Azure Subscription Id"
}
# Azure Client Id/appId
variable "azure-client-id" {
    type = string
    description = "Azure Client Id/appId"
}
# Azure Client Secret
variable "azure-client-secret" {
    type = string
    description = "Azure Client Secret"
}
# Azure Tenant Id
variable "azure-tenant-id" {
    type = string
    description = "Azure Tenant Id"
}

Next, in a location that you will never, ever make public, you can place your terraform.tfvars file which will contain the values for your credential variables.

# Azure Subscription Id
azure-subscription-id = "DUMMY"
# Azure Client Id/appId
azure-client-id = "DUMMY"
# Azure Client Secret/password
azure-client-secret = "DUMMY"
# Azure Tenant Id
azure-tenant-id = "DUMMY"

The final piece that pulls all this together is the Terraform main.tf file which needs information about the variables you have defined. The provider resource for “azurerm” will need to be updated with the credentials:

provider "azurerm" {
   subscription_id  = var.azure-subscription-id
   client_id        = var.azure-client-id
   client_secret    = var.azure-client-secret
   tenant_id        = var.azure-tenant-id
   version = "~>2.0"
   features {}
}

The State File

Terraform keeps track of the state of our infrastructure and services by using a state file. The state file can of course be local, but for working within a team and to ensure the resiliency of this important file, it can be stored in a remote, shared location. There are a number of options available for remote, secure, shared storage. With AWS, for example, S3 could be a good choice. We will look at storing the file in an Azure storage account.

There is a tutorial on docs.microsoft.com that details how you might set up Azure storage as your state file backend. The state file can be stored in a new resource group and storage account within your existing Azure infrastructure. An additional benefit of using an Azure storage resource as a backend is that with encryption enabled, the persisted state file is encrypted at rest, decrypted when needed by Terraform and used, with no local, clear-text storage of the state file. This is important because the state file can contain sensitive data about the resources that Terraform manages, including access credentials.

See Tutorial - Store Terraform state in Azure Storage | Microsoft Docs for details on creating the backend storage required. The first step is to configure the storage account and container using the script provided in the tutorial.

#!/bin/bash

RESOURCE_GROUP_NAME=terraformstate-rg
STORAGE_ACCOUNT_NAME=terraformstate$RANDOM
CONTAINER_NAME=terraform-state

# Create resource group
az group create --name $RESOURCE_GROUP_NAME --location eastus

# Create storage account
az storage account create --resource-group $RESOURCE_GROUP_NAME --name $STORAGE_ACCOUNT_NAME --sku Standard_LRS --encryption-services blob

# Get storage account key
ACCOUNT_KEY=$(az storage account keys list --resource-group $RESOURCE_GROUP_NAME --account-name $STORAGE_ACCOUNT_NAME --query '[0].value' -o tsv)

# Create blob container
az storage container create --name $CONTAINER_NAME --account-name $STORAGE_ACCOUNT_NAME --account-key $ACCOUNT_KEY

echo "storage_account_name: $STORAGE_ACCOUNT_NAME"
echo "container_name: $CONTAINER_NAME"
echo "access_key: $ACCOUNT_KEY"

The successful output from that script should be a JSON export of the resource and a confirmation of the details as echoed by the script itself, below you can see the last few lines of a successful run.

{
  "created": true
}
storage_account_name: terraformstate14106
container_name: terraform-state
access_key: ahugelonglistofcharactersthatrepresentyouruniquestorageaccesskeyiwasntgoingtoleavetherealkeyhere==

Once the script has successfully run, you should have a new storage account and container for your Terraform state file.

To use this access key within Terraform we could use an environment variable, ARM_ACCESS_KEY to hold the value. This is achieved with “export ARM_ACCESS_KEY=.

However, what is more secure is the creation of an Azure Key Vault to hold the secret access key.

Again, turning to docs.microsoft.com, the Microsoft documentation shows how this can be done at the link below:

Quickstart: Set and retrieve a secret from Azure Key Vault | Microsoft Docs

We will show here an example using the Azure CLI. First, the creation of a new key vault to hold our secrets:

$ az keyvault create --name mykeyvault01 --resource-group keyvault-rg --location uksouth

Next, placing our key/value secret in the vault. The secret key used is the access_key for the storage from the previous example.

$ az keyvault secret set --vault-name mykeyvault01 --name terraform-backend-key --value "<secret key here>”

We can test retrieval of the key as follows:

$ az keyvault secret show --name terraform-backend-key --vault-name mykeyvault01 --query value -o tsv

Which will return the key value.

Now, when we need to use Terraform and we want to export our environment variable to contain the storage access key we can do so by populating our environment variable directly with the returned value.

$ export ARM_ACCESS_KEY=$( az keyvault secret show --name terraform-backend-key --vault-name mykeyvault01 --query value -o tsv )

The benefit of this is that within scripts, or at the command line, the key is never stored or revealed in plain text on the console.

Configuring the back end

The Terraform state back end storage is now configured, but you must run the “terraform init” command in order to initialize the working directory and the configuration files for Terraform,

As part of the initialisation, Terraform will attempt to create the first Terraform state file for your stack.

Before we do this, there are a few remaining items to place into the main.tf file in order to tell Terraform about your backend storage and how to find the location for the state file:

Firstly, the terraform resource itself.

terraform {
  required_version = ">= 0.12"
  backend "azurerm" {
    resource_group_name    = "terraformstate-rg"
    storage_account_name   = "terraformstate14107"
    container_name         = "terraform-state"
    key                    = "terraform.tfstate"
    }
}

As you can see, it defines the resource group, storage account and container name in which to find the backend storage.

Then, as a reminder, the azurerm provider additionally details the Azure subscription ID and service principal details:

provider "azurerm" {
   subscription_id  = var.azure-subscription-id
   client_id        = var.azure-client-id
   client_secret    = var.azure-client-secret
   tenant_id        = var.azure-tenant-id
   version = "~>2.0"
   features {}
}

To test this all out we can create a simple resource group using the Terraform resource type “azurerm_resource_group”. This requires two parameters before the resource block; a name of the Terraform resource itself and an optional variable name for the resource group it will create.

Within the block we can define the location and any tags that we wish to attach to the new resource. For example:

resource "azurerm_resource_group" "demo-terraform-rg" {
   name = "egg-demo-terraform-rg"
   location = "uksouth"
   tags = {
    environment = "demo terraform RG"
   }
}

Of course, we can do much more than create resource groups. For our first pass, I will create a virtual network and a contained subnet.

  • Resource Group : egg-demo-terraform-rg
  • Virtual Network : tfvnet01
  • Address space: 10.10.0.0/16
  • Subnet: private_subnet01
  • Address prefix: 10.10.2.0/24

The first step is to run “terraform init”:

$ terraform init

Initializing the backend...

Successfully configured the backend "azurerm"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "azurerm" (hashicorp/azurerm) 2.44.0...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work.

If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary.

With that complete, we are ready to run “terraform plan”. The terraform plan command will look at both the state of our currently configured resources and the desired state that has been declared in our Terraform template files. It will then detail the changes that are required (the plan) to move to the desired state. There will be no actual changes to our infrastructure by running the plan command, but we will get a summary of the required changes.

Let us see how we fare...

$ terraform plan
Acquiring state lock. This may take a few moments...
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # azurerm_resource_group.demo-terraform-rg will be created
  + resource "azurerm_resource_group" "demo-terraform-rg" {
      + id       = (known after apply)
      + location = "uksouth"
      + name     = "egg-demo-terraform-rg"
      + tags     = {
          + "environment" = "demo terraform RG"
        }
    }

  # azurerm_subnet.tfsubnet01 will be created
  + resource "azurerm_subnet" "tfsubnet01" {
      + address_prefix                                 = (known after apply)
      + address_prefixes                               = [
          + "10.10.1.0/24",
        ]
      + enforce_private_link_endpoint_network_policies = false
      + enforce_private_link_service_network_policies  = false
      + id                                             = (known after apply)
      + name                                           = "private_subnet01"
      + resource_group_name                            = "egg-demo-terraform-rg"
      + virtual_network_name                           = "tfvnet01"
    }

  # azurerm_virtual_network.tfvnet01 will be created
  + resource "azurerm_virtual_network" "tfvnet01" {
      + address_space         = [
          + "10.10.0.0/16",
        ]
      + guid                  = (known after apply)
      + id                    = (known after apply)
      + location              = "uksouth"
      + name                  = "tfvnet01"
      + resource_group_name   = "egg-demo-terraform-rg"
      + subnet                = (known after apply)
      + vm_protection_enabled = false
    }

Plan: 3 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

As you can see from the output, Terraform has identified the resources it needs to create for us. We can review the plan and if we are happy, we are now ready to run “terraform apply” which will create our new resources for us.

$ terraform apply
Acquiring state lock. This may take a few moments...

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # azurerm_resource_group.demo-terraform-rg will be created
  + resource "azurerm_resource_group" "demo-terraform-rg" {
      + id       = (known after apply)
      + location = "uksouth"
      + name     = "egg-demo-terraform-rg"
      + tags     = {
          + "environment" = "demo terraform RG"
        }
    }

  # azurerm_subnet.tfsubnet01 will be created
  + resource "azurerm_subnet" "tfsubnet01" {
      + address_prefix                                 = (known after apply)
      + address_prefixes                               = [
          + "10.10.1.0/24",
        ]
      + enforce_private_link_endpoint_network_policies = false
      + enforce_private_link_service_network_policies  = false
      + id                                             = (known after apply)
      + name                                           = "private_subnet01"
      + resource_group_name                            = "egg-demo-terraform-rg"
      + virtual_network_name                           = "tfvnet01"
    }

  # azurerm_virtual_network.tfvnet01 will be created
  + resource "azurerm_virtual_network" "tfvnet01" {
      + address_space         = [
          + "10.10.0.0/16",
        ]
      + guid                  = (known after apply)
      + id                    = (known after apply)
      + location              = "uksouth"
      + name                  = "tfvnet01"
      + resource_group_name   = "egg-demo-terraform-rg"
      + subnet                = (known after apply)
      + vm_protection_enabled = false
    }

Plan: 3 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

We are asked to review and confirm the changes, and if we do, terraform will take just a few seconds to create our resources.

azurerm_resource_group.demo-terraform-rg: Creating...
azurerm_resource_group.demo-terraform-rg: Creation complete after 0s [id=/subscriptions/22aa5fd8-b1e0-43d2-a4f7-21ad2c555339/resourceGroups/egg-demo-terraform-rg]
azurerm_virtual_network.tfvnet01: Creating...
azurerm_virtual_network.tfvnet01: Creation complete after 4s [id=/subscriptions/22aa5fd8-b1e0-43d2-a4f7-21ad2c555339/resourceGroups/egg-demo-terraform-rg/providers/Microsoft.Network/virtualNetworks/tfvnet01]
azurerm_subnet.tfsubnet01: Creating...
azurerm_subnet.tfsubnet01: Creation complete after 4s [id=/subscriptions/22aa5fd8-b1e0-43d2-a4f7-21ad2c555339/resourceGroups/egg-demo-terraform-rg/providers/Microsoft.Network/virtualNetworks/tfvnet01/subnets/private_subnet01]

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Sure enough, we have a new resource group with a new vnet and subnet available:

$ az network vnet list -g egg-demo-terraform-rg --output table
Name      ResourceGroup          Location    NumSubnets    Prefixes     
--------  ---------------------  ----------  ------------  ------------   
tfvnet01  egg-demo-terraform-rg  uksouth     1             10.10.0.0/16

Modifying existing resources.

Because Terraform is stateful, we are able to not just create resources, but to modify existing resources within the current state as well. For example, I missed the tags off the vnet and it would be nice to add the same tag that I placed on the resource group. All I need to do is add the tags to the appropriate resource blocks and re-run the plan/apply verbs of Terraform.

In the example below, I have actually changed the value of the environment tag on the resource group as well, from “demo terraform RG” to demo terraform stack”. This means we are expecting Terraform to update an existing tag and to add tags to an existing resource.

 tags = {
      environment   = "demo terraform stack"
   }

This time, Terraform informs me that there are 2 items to change, none to create or destroy:

Plan: 0 to add, 2 to change, 0 to destroy.

Note the tilde for the first change, the tag for the resource group will change:

      ~ tags     = {
          ~ "environment" = "demo terraform RG" -> "demo terraform stack"
        }

While we have a green plus to indicate that the tag for the vnet will be added:

   ~ tags                  = {
          + "environment" = "demo terraform stack"
        }

We run the apply and Success!

Apply complete! Resources: 0 added, 2 changed, 0 destroyed.

If we now check the tags for the resource group and resource by looking for entities that match our new tag, we see that all was completed as expected.

$ az group list --tag 'Environment=demo terraform stack' --output table
Name                   Location    Status
---------------------  ----------  ---------
egg-demo-terraform-rg  uksouth     Succeeded

$ az resource list --tag 'Environment=demo terraform stack' --output table
Name      ResourceGroup          Location    Type                               
--------  ---------------------  ----------  ---------------------------------  
tfvnet01  egg-demo-terraform-rg  uksouth     Microsoft.Network/virtualNetworks

Destroy

Finally, how to remove the resources you have created if you don’t need them anymore? The Terraform function for this is, unsurprisingly, called ‘destroy’. Use it with caution. Please note that it won’t delete anything that is not defined within your Terraform state for the stack. Here it is in action, removing our resource group, virtual network and subnet.

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # azurerm_resource_group.demo-terraform-rg will be destroyed
  - resource "azurerm_resource_group" "demo-terraform-rg" {
      - id       = "/subscriptions/22aa5fd8-b1e0-43d2-a4f7-21ad2c555339/resourceGroups/egg-demo-terraform-rg" -> null
      - location = "uksouth" -> null
      - name     = "egg-demo-terraform-rg" -> null
      - tags     = {
          - "environment" = "demo terraform stack"
        } -> null
    }

  # azurerm_subnet.tfsubnet01 will be destroyed
  - resource "azurerm_subnet" "tfsubnet01" {
      - address_prefix                                 = "10.10.1.0/24" -> null
      - address_prefixes                               = [
          - "10.10.1.0/24",
        ] -> null
      - enforce_private_link_endpoint_network_policies = false -> null
      - enforce_private_link_service_network_policies  = false -> null
      - id                                             = "/subscriptions/22aa5fd8-b1e0-43d2-a4f7-21ad2c555339/resourceGroups/egg-demo-terraform-rg/providers/Microsoft.Network/virtualNetworks/tfvnet01/subnets/private_subnet01" -> null
      - name                                           = "private_subnet01" -> null
      - resource_group_name                            = "egg-demo-terraform-rg" -> null
      - service_endpoint_policy_ids                    = [] -> null
      - service_endpoints                              = [] -> null
      - virtual_network_name                           = "tfvnet01" -> null
    }

  # azurerm_virtual_network.tfvnet01 will be destroyed
  - resource "azurerm_virtual_network" "tfvnet01" {
      - address_space         = [
          - "10.10.0.0/16",
        ] -> null
      - dns_servers           = [] -> null
      - guid                  = "eac7b16c-f385-4ed3-8d98-7972df7a2f8f" -> null
      - id                    = "/subscriptions/22aa5fd8-b1e0-43d2-a4f7-21ad2c555339/resourceGroups/egg-demo-terraform-rg/providers/Microsoft.Network/virtualNetworks/tfvnet01" -> null
      - location              = "uksouth" -> null
      - name                  = "tfvnet01" -> null
      - resource_group_name   = "egg-demo-terraform-rg" -> null
      - subnet                = [
          - {
              - address_prefix = "10.10.1.0/24"
              - id             = "/subscriptions/22aa5fd8-b1e0-43d2-a4f7-21ad2c555339/resourceGroups/egg-demo-terraform-rg/providers/Microsoft.Network/virtualNetworks/tfvnet01/subnets/private_subnet01"
              - name           = "private_subnet01"
              - security_group = ""
            },
        ] -> null
      - tags                  = {
          - "environment" = "demo terraform stack"
        } -> null
      - vm_protection_enabled = false -> null
    }

Plan: 0 to add, 0 to change, 3 to destroy.

All gone!

azurerm_resource_group.demo-terraform-rg: Destruction complete after 45s

Destroy complete! Resources: 3 destroyed.

Summary

This short introduction to Terraform with Azure has shown us how to install Terraform and get setup with a basic environment which will connect to our Azure account using a service principal to avoid manual password entry. Terraform is able to access our Azure subscription in a secure manner by using the details of this service principal.

The state file is vital to Terraform and it may contain sensitive information about your infrastructure. For these reasons we must protect it and we have seen how it can be shared securely in a remote location using Azure storage accounts and an access key stored within Azure Key Vault.

We barely scratched the surface in this article when we went on to configure a basic resource group with a new virtual network and contained subnet. Terraform is much more powerful and has hundreds of resources with the azurerm provider to manage all aspects of your Azure cloud infrastructure.

This has been my first ever blog post, I hope you liked it.