Terraform for Infrastructure Automation

Friday 17 March 2017 Terraform, AWS, Azure Comments

Before starting to learn Terraform, you first need to learn certain concept in the modern infrastructure. By the adoption of cloud technologies, the amount of servers used by almost any project is growing rapidlly. As a result, managing and configuring IT infrastructure with traditional ways by manual approach is definitely not going to play out well and become less and less relevant.

No surprises, companies of any size, from small start-ups to huge entreprises, are adopting new techniques and tools to manage and automate their infrastructures called Infrastructure as Code (IaC). This new approach is a foundation for DevOps culture because both operations and developers approach their work in the same way.

Terraform is an open source utility, created by the HashiCorp company, the same company that created Vagrant, Packer, Consul and other popular infrastructure tools. It was initially released in July 2014, and since then, has come a long way to become one of the most important tools for infrastructure provisioning and management.

To learn how exactly it works, this tutorial shows how to use Terraform to create secure, private, site-to-site connections between Amazon Web services (AWS) VPC and Microsoft Azure virtual network, using virtual private networks (VPNs) Tunnel. This is a multi-cloud deployment.

befor we start you need to install Terraform by following the simple steps from the install web page Getting Started.

Deployment architecture

In this tutorial, you build the following deployment environment.

This tutorial guides you through:

  • Building custom VPC networks with user-specified CIDR blocks in AWS and Azure.
  • Creating virtual private gateway (VPG), customer gateway and a VPN connection in AWS.
  • Deploying a VM instance in each virtual network.


Azure

The first step is to deploy the Azure infrastructure. we create an empty directory that will hold the new project.

Terraform templates

First thing we do is tell Terraform which provider we are going to use and setup all variables to use in this project. We need to create azure-provider.tfvars and variables.tf files.

In azure-provider.tfvars file, we define the default values for the variables.

tenant_id="your azure cred tenant id"
client_id="your azure cred client id"
client_secret="your azure client secret"
subscription_id="your azure subscription"
region="westeurope"
resource_group_name="rgdemo"
vnet_name="vnetdemo"
cidr_block="10.0.0.0/16"
subnet_name="subnetdemo"
subnet_cidr_block="10.0.0.0/24"
vm_username="azureuser"
public_key="your key contant"


The variables.tf file containes all essential variables needed to build the azure infrastructure. With the default values in place, Terraform won't ask for value interactively anymore. It will pick default value unless other sources of variable are present. For information the are three types of variables you can set:

  • the string variables (default ones)
  • the map variable
  • the list variables


You can only interactively set the string variables; for map and list, you have to use other methodes.

variable "client_id" {
description = "Access key for Azure"
}
variable "client_secret" {
description = "Secret Key for Azure"
}
variable "subscription_id" {
description = "Subscription ID for Azure"
}
variable "tenant_id" {
description = "Tenant ID; from EndPoint in classic panel .."
}
provider "azurerm" {
subscription_id = "${var.subscription_id}"
client_id = "${var.client_id}"
client_secret = "${var.client_secret}"
tenant_id = "${var.tenant_id}"
}
variable "region" {
description = "The default Azure region for the resource provisioning"
}
variable "resource_group_name" {
description = "Resource group name that will contain various resources"
}
variable "cidr_block" {
description = "CIDR block for Virtual Network"
}
variable "subnet_cidr_block" {
description = "CIDR block for Subnet within a Virtual Network"
}
variable "subnet_name" {
description = "Name for Subnet within a Virtual Network"
}
variable "vm_username" {
description = "Enter admin username to SSH into Linux VM"
}
variable "public_key" {
description = "The public key content"
}
variable "vnet_name" {
description = "The virtual network name"
}


To enable SSH onto our Azure instance (VM) we need to create a key pair and add it to Terraform setup through the public_key variable.

$ssh-keygen -t rsa -b 4096 -C "your_email@example.com" -f $HOME/.ssh/your_key_name
$cat $HOME/.ssh/your_key_name.pub | pbcopy


After that we will proceed to describe the Azure resources in single template main.tf. Eeach section of this terraform template describe a resource to provision on Microsoft Azure. We will describe the following resource:

  • Resource group
  • virtual network and a subnet
  • public IP address
  • Network Security Group
  • Virtual network interface card
  • Storage account for diagnostics
  • Virtual machine
resource "azurerm_resource_group" "rgroup" {
name = "${var.resource_group_name}"
location = "${var.region}"
 
tags {
environment = "Terraform Demo"
}
}
 
resource "azurerm_virtual_network" "aznetwork" {
name = "${var.vnet_name}"
address_space = ["${var.cidr_block}"]
location = "${var.region}"
resource_group_name = "${azurerm_resource_group.rgroup.name}"
 
tags {
environment = "Terraform Demo"
}
}
 
resource "azurerm_subnet" "azsubnet" {
name = "${var.subnet_name}"
resource_group_name = "${azurerm_resource_group.rgroup.name}"
virtual_network_name = "${azurerm_virtual_network.aznetwork.name}"
address_prefix = "${var.subnet_cidr_block}"
}
 
resource "azurerm_public_ip" "azpublicip" {
name = "myPublicIP"
location = "${var.region}"
resource_group_name = "${azurerm_resource_group.rgroup.name}"
public_ip_address_allocation = "dynamic"
 
tags {
environment = "Terraform Demo"
}
}
 
resource "azurerm_network_security_group" "azpublicipnsg" {
name = "myNetworkSecurityGroup"
location = "${var.region}"
resource_group_name = "${azurerm_resource_group.rgroup.name}"
 
security_rule {
name = "SSH"
priority = 1001
direction = "Inbound"
access = "Allow"
protocol = "Tcp"
source_port_range = "*"
destination_port_range = "22"
source_address_prefix = "*"
destination_address_prefix = "*"
}
 
tags {
environment = "Terraform Demo"
}
}
 
resource "azurerm_network_interface" "aznic" {
name = "myNIC"
location = "${var.region}"
resource_group_name = "${azurerm_resource_group.rgroup.name}"
 
ip_configuration {
name = "myNicConfiguration"
subnet_id = "${azurerm_subnet.azsubnet.id}"
private_ip_address_allocation = "dynamic"
public_ip_address_id = "${azurerm_public_ip.azpublicip.id}"
}
 
tags {
environment = "Terraform Demo"
}
}
 
resource "random_id" "randomId" {
keepers = {
# Generate a new ID only when a new resource group is defined
resource_group = "${azurerm_resource_group.rgroup.name}"
}
 
byte_length = 8
}
 
resource "azurerm_storage_account" "azstorageaccount" {
name = "diag${random_id.randomId.hex}"
resource_group_name = "${azurerm_resource_group.rgroup.name}"
location = "${var.region}"
account_replication_type = "LRS"
account_tier = "Standard"
 
tags {
environment = "Terraform Demo"
}
}
 
resource "azurerm_virtual_machine" "azvm" {
name = "myVM"
location = "${var.region}"
resource_group_name = "${azurerm_resource_group.rgroup.name}"
network_interface_ids = ["${azurerm_network_interface.aznic.id}"]
vm_size = "Standard_DS1_v2"
 
storage_os_disk {
name = "myOsDisk"
caching = "ReadWrite"
create_option = "FromImage"
managed_disk_type = "Premium_LRS"
}
 
storage_image_reference {
publisher = "Canonical"
offer = "UbuntuServer"
sku = "16.04.0-LTS"
version = "latest"
}
 
os_profile {
computer_name = "myvm"
admin_username = "${var.vm_username}"
}
 
os_profile_linux_config {
disable_password_authentication = true
ssh_keys {
path = "/home/azureuser/.ssh/authorized_keys"
key_data = "${var.public_key}"
}
}
 
boot_diagnostics {
enabled = "true"
storage_uri = "${azurerm_storage_account.azstorageaccount.primary_blob_endpoint}"
}
 
tags {
environment = "Terraform Demo"
}
}


Then we will define the project outputs in a single file outputs.tf

output "Public Ip" {
value = "${azurerm_public_ip.azpublicip.ip_address}"
}

The full project is avalable in my git repository.

Build and deploy the infrastructure

With your Terraform template created, the first step is to initialize Terraform. This step ensures that Terraform has all the prerequisites to build your template in Azure.

$ terraform init


Use the Terraform validate command to validate the syntax of your configuration files. This validation check is simpler than those performed as part of the plan and apply commands in subsequent steps. The validate command does not authenticate with any providers.

$ terraform validate -var-file=./azure-provider.tfvars


If you don't see an error message, you have completed an initial validation of your file syntax and basic semantics. If you do see an error message, validation failed.

Use the Terraform plan command to review the deployment without instantiating resources in the cloud. The plan command requires successful authentication with all providers specified in the configuration.

$ terraform plan -var-file=./azure-provider.tfvars


The plan command returns an output listing of resources to be added, removed, or updated. The last line of the plan output shows a count of resources to be added, changed, or destroyed.

Optionally, visualize your resource dependencies by using the Terraform graph command. The dependency graph helps you analyze your deployed resources. If you have the GraphViz package installed, You can view a previously prepared version of the output file at azurevnet_plan_graph.png by running run_graph.sh script or with the commande terraform graph | dot -Tpng > azurevnet_plan_graph.png creates the PNG file azurevnet_plan_graph.png, which looks similar to the following:

If everything looks correct and you are ready to build the infrastructure in Azure, apply the template in Terraform:

$ terraform apply -var-file=./azure-provider.tfvars

AWS

The second step is to describe the AWS infrastructure under a new directory. The full project is defined in this repository.

Terraform templates

Under aws-provider.tfvars and variables.tf files descibe all essential variables for this project.

region = "eu-west-1"
access_key="your aws access key"
secret_key="your aws secret key"
name="vpcdemo"
cidr_block="192.168.0.0/16"
zones="eu-west-1a,eu-west-1b,eu-west-1c"
private_subnets="192.168.0.0/24"
vpn_bgp_asn="65000"
vpn_ip_address="xxx.xxx.xxx.xxx"
vpn_dest_cidr_block="10.0.0.0/16"
aws_instance_type="t2.nano"
aws_ami="ami-58d7e821"
public_key="your key pair content"
key_name="phkey"


provider "aws" {
region= "${var.region}"
access_key = "${var.access_key}"
secret_key = "${var.secret_key}"
}
variable "region" {
description = "The aws region"
}
variable "access_key" {
description = "The aws access_key"
}
variable "secret_key" {
description = "The aws secret_key"
}
variable "vpn_bgp_asn" {
description = "BPG Autonomous System Number (ASN) of the customer gateway for a dynamically routed VPN connection."
}
variable "vpn_ip_address" {
description = "Internet-routable IP address of the customer gateway's external interface."
}
variable "vpn_dest_cidr_block" {
description = "Internal network IP range to advertise over the VPN connection to the VPC."
}
variable "name" {
description = "The name of the VPC."
}
variable "cidr_block" {
description = "The CIDR block for the VPC."
}
variable "zones" {
description = "List of availability zones to use."
}
variable "public_subnets" {
description = "List of CIDR blocks to use as public subnets; instances launced will be assigned a public IP address."
default = ""
}
variable "private_subnets" {
description = "List of CIDR blocks to use as private subnets; instances launced will NOT be assigned a public IP address."
default = ""
}
variable "enable_dns_hostnames" {
description = "Enable DNS hostnames in the VPC (default false)."
default = false
}
variable "enable_dns_support" {
description = "Enable DNS support in the VPC (default true)."
default = true
}
variable "key_name" {
description = "keypair name"
}
variable "public_key" {
description = "The public key content"
}
variable "aws_ami" {
description = "The aws ami."
}
variable "aws_instance_type" {
description = "The aws instance type."
}

The main template descibes the following resources:

  • VPC and subnet
  • Virtual private gateway
  • Customer gateway
  • VPN connection
  • Routes
  • Security group and rule
  • Instance (VM)
resource "aws_vpc" "vpc" {
cidr_block = "${var.cidr_block}"
enable_dns_support = "${var.enable_dns_support}"
enable_dns_hostnames = "${var.enable_dns_hostnames}"
tags {
Name = "${var.name}-vpc"
}
}
 
resource "aws_subnet" "private_subnet" {
vpc_id = "${aws_vpc.vpc.id}"
cidr_block = "${element(split(",", var.private_subnets), count.index)}"
availability_zone = "${element(split(",", var.zones), count.index)}"
count = "${length(compact(split(",", var.private_subnets)))}"
tags {
Name = "${format("%s-private-%d", var.name, count.index + 1)}"
}
}
 
resource "aws_internet_gateway" "vpc-igw" {
vpc_id = "${aws_vpc.vpc.id}"
tags {
Name = "${var.name}-igw"
}
}
 
resource "aws_vpn_gateway" "vpc-vgw" {
vpc_id = "${aws_vpc.vpc.id}"
tags {
Name = "${var.name}-vgw"
}
}
 
resource "aws_route" "vgw-route" {
route_table_id = "${aws_vpc.vpc.main_route_table_id}"
destination_cidr_block = "${var.vpn_dest_cidr_block}"
gateway_id = "${aws_vpn_gateway.vpc-vgw.id}"
}
 
resource "aws_customer_gateway" "vpc-cgw" {
bgp_asn = "${var.vpn_bgp_asn}"
ip_address = "${var.vpn_ip_address}"
type = "ipsec.1"
tags {
Name = "${var.name}-cgw"
}
}
 
resource "aws_vpn_connection" "vpc-vpn" {
vpn_gateway_id = "${aws_vpn_gateway.vpc-vgw.id}"
customer_gateway_id = "${aws_customer_gateway.vpc-cgw.id}"
type = "ipsec.1"
static_routes_only = true
tags {
Name = "${var.name}-vpn"
}
}
 
resource "aws_vpn_gateway_route_propagation" "vpnroute" {
vpn_gateway_id = "${aws_vpn_gateway.vpc-vgw.id}"
route_table_id = "${aws_vpc.vpc.main_route_table_id}"
}
 
resource "aws_key_pair" "vm-key" {
key_name = "${var.key_name}"
public_key = "${var.public_key}"
}
 
resource "aws_security_group" "vm-sg" {
name = "vm-security-group"
vpc_id = "${aws_vpc.vpc.id}"
 
ingress {
protocol = "tcp"
from_port = 22
to_port = 22
cidr_blocks = ["0.0.0.0/0"]
}
}
 
resource "aws_instance" "vm-aws" {
ami = "${var.aws_ami}"
instance_type = "${var.aws_instance_type}"
subnet_id = "${aws_subnet.private_subnet.id}"
key_name = "${aws_key_pair.vm-key.key_name}"
 
associate_public_ip_address = false
 
vpc_security_group_ids = [
"${aws_security_group.vm-sg.id}"
]
 
tags {
Name = "aws-vm-test"
}
}


The ouput variables are defined under ouputs.tf file.

output "VPC" {
value = "${aws_vpc.vpc.id}"
}
output "Private subnets" {
value = "${join(",", aws_subnet.private_subnet.*.id)}"
}
output "Internet gateway" {
value = "${aws_internet_gateway.vpc-igw.id}"
}
output "Customer gateway" {
value = "${aws_customer_gateway.vpc-cgw.id}"
}
output "VPN gateway" {
value = "${aws_vpn_gateway.vpc-vgw.id}"
}
output "VPN gateway config" {
value = "${aws_vpn_connection.vpc-vpn.customer_gateway_configuration}"
}


Build and deploy the infrastructure

Now that you’ve created your template in HCL, let’s initialize Terraform and ask it to apply the configuration to AWS.

$ terraform init


With the AWS provider installed, let’s ask Terraform what changes it’s planning to make to your AWS account so it matches the configuration you previously defined:

$ terraform plan -var-file=./aws-provider.tfvars


To visualize your resource dependencies by using the Terraform graph command included in script.

$ terraform plan -var-file=./aws-provider.tfvars

The result is shown in the following image:

The plan command is important, as it allows you to preview the changes for accuracy before actually making them. Once you’re comfortable with the execution plan, it’s time to apply it: To visualize your resource dependencies by using the Terraform graph command included in script.

$ terraform apply -var-file=./aws-provider.tfvars


Use the Terraform show command to inspect the deployed resources and verify the current state.

$ terraform show


The terraform state list command will list all resources in the state file

terraform state list
aws_customer_gateway.vpc-cgw
aws_instance.vm-aws
aws_internet_gateway.vpc-igw
aws_key_pair.vm-key
aws_route.vgw-route
aws_security_group.vm-sg
aws_subnet.private_subnet
aws_vpc.vpc
aws_vpn_connection.vpc-vpn
aws_vpn_gateway.vpc-vgw
aws_vpn_gateway_route_propagation.vpnroute


If, later in your workflow, you want to redisplay the output variable values, use the output command.

$ terraform output


Configure IPSec

To get the VPN connection we need to ssh to the azure instance host

$ ssh azureuser@{vm host}


Edit the IPSec files

$ sudo vi/etc/ipsec.conf


Then Replace TUNNEL_1_IP_ADDRESS and TUNNEL_2_IP_ADDRESS with the IP addresses for each tunnel that you copied from AWS terraform outputs.

# ipsec.conf - strongSwan IPsec configuration file
 
# basic configuration
 
config setup

conn %default
esp=aes128-sha1-modp1024
ikelifetime=28800s
keylife=3600s
rekeymargin=3m
keyingtries=%forever
keyexchange=ikev1
mobike=no
left=%any
leftsubnet=10.0.0.0/16 # Azure CIDR
dpdaction=restart
auto=start
authby=secret
rightsubnet=192.168.0.0/16 # AWS CIDR
 
conn VPC-CUST-GW1
right=xxx.xxx.xxx.xxx #TUNNEL_1_IP_ADDRESS
conn VPC-CUST-GW2
right=xxx.xxx.xxx.xxx #TUNNEL_2_IP_ADDRESS


Edit the second ipsec file.

$ sudo vi/etc/ipsec.secrets


Replace TUNNEL_1_IP_ADDRESS  and  TUNNEL_2_IP_ADDRESS  with the IP addresses for Tunnels 1 and 2. Replace  TUNNEL_1_PRESHARED_KEY  and  TUNNEL_2_PRESHARED_KEY with the values you copied from the outputs Configuration.

Example of final text

#This file holds shared secrets or RSA private keys for authentication.
# RSA private key for this host, authenticating it to any other host
# which knows the public part.
 
34.247.15.26 : PSK "8ytgXDDApcUztcDEpOLp1uJQZJuiC"
52.9.89.129 : PSK "XNYlud.GsMc62wZI9kG6P_SB8uw2Y"

To start the IPSec services run the following two commands:
$ sudo ipsec restart
$ sudo ipsec status

IPSec should return a message similar to the following:

Security Associations (2 up, 0 connecting):
VPC-CUST-GW2[2]: ESTABLISHED 3 seconds ago, 10.0.0.4[10.0.0.4]...52.49.89.129[52.49.89.129]
VPC-CUST-GW2{1}: INSTALLED, TUNNEL, reqid 1, ESP in UDP SPIs: c83198d7_i 3f3c9004_o
VPC-CUST-GW2{1}: 10.0.0.0/16 === 192.168.0.0/16
VPC-CUST-GW1[1]: ESTABLISHED 3 seconds ago, 10.0.0.4[10.0.0.4]...34.247.5.26[34.247.5.26]
VPC-CUST-GW1{2}: INSTALLED, TUNNEL, reqid 1, ESP in UDP SPIs: c57b5dd7_i b79b41a8_o
VPC-CUST-GW1{2}: 10.0.0.0/16 === 192.168.0.0/16


If you check your AWS VPN connection tunnels, they should be on UP status.

You have successfully deployed a secure, private, site-to-site connection between AWS and Azure using VPN.

Cleaning up

Clean up the deployed resources. You will continue to be billed for your resources until you run the destroy deployment command. Run the optional plan -destroy command to review the resources affected by destroy:

$ terraform plan -var-file=./azure-provider.tfvars -destroy
$ terraform plan -var-file=./aws-provider.tfvars -destroy


Because the destroy command will permanently delete your resources, you must confirm your intentions by entering yes.

$ terraform destroy -var-file=./azure-provider.tfvars
$ terraform destroy -var-file=./aws-provider.tfvars