Terraform Overview

Categories: Cloud, Infrastructure

Terraform for IaaS

Terraform is a tool for configuring Infrastructure-as-a-Service (IaaS) cloud environments. The core of Terraform is open-source; its primary developer (Hashicorp) also offers proprietary components on top but the core (free) components are useful on their own.

A recent article here discussed what is a cloud, and described how an IaaS cloud environment provides a REST API for managing resources such as networks, storage volumes and virtual machines. As well as the API, a cloud provider usually also provides a web-based GUI for managing resources. However defining large numbers of such resources interactively over a UI is slow, error-prone, and not repeatable; it is better to automate such work via the cloud-provider’s API.

I just recently attended a presentation on Terraform (thanks Eric), and this is a brief summary of what I learned from that presentation. Obviously I am no expert - this is just an overview.

Terraform is a simple application which is run manually on a laptop or server, or executed from a CI tool like Jenkins. When executed, it:

  • reads an “infrastructure definition” file which defines what resources (VMs, volumes, and networks should exist);
  • makes calls to a cloud provider API in order to determine what resources already exist;
  • generates a human-readable “plan” describing the changes it intends to apply in order to bring the existing cloud configuration “in-sync” with the definition;
  • optionally waits for the “plan” to be approved by an operator; then
  • makes calls to the cloud provider API to make the modifications described in the plan (ie creates/deletes/modifies resources as needed).

The result is that the cloud resources are brought into the state described by the input infrastructure definition. These definitions can (and should) be stored in a version-control system. A basic JSON syntax is supported, and a more advanced syntax supporting multiple files, templates, and other useful features.

Unfortunately, there are many different cloud-management APIs - Google GCP, Amazon AWS, OpenStack, CloudStack, and more. Terraform is therefore divided into a “core” module, and an adapter (called a “provider” by terraform) per supported API.

Obviously, different cloud providers also have different features. Terraform’s config syntax does not attempt to standardize definitions of features that are common across multiple cloud providers; they are apparently just too different. However because the principles behind the engine are the same, porting a configuration definition for one cloud provider to another is less work than moving from one cloud-specific server provisioning framework to another.

Note that it it not part of Terraform’s feature-set to install software on virtual machines; it simply defines virtual machines (referencing an existing VM image file) and starts/stops them. While slightly off-topic, good options for building VM images are packer (also from Hashicorp), Chef and Puppet.

Terraform State

For each “environment” (terraform config file), Terraform maintains a database of the resources it is managing. Resources include VMs, disks, network definitions, and many other possibilities. Each resource effectively has two keys: the name it is known as in the Terraform config file, and the id allocated to the resource by the cloud provider. Terraform fetches the state of each resource registered in its database from the cloud before making updates, ie it can handle changes made in the cloud such as deletion or modification of resources. It does not detect resources added to the cloud, or rather does not care - it only manages those resources in its database. An “import” operation can register an existing resource from the cloud in the state if desired, but this is only useful as a preparatory step to adding an entry in the config-file - or in preparation for deleting all known resources.

State is stored by default in a local file. This obviously has implications for reliability (losing the file is bad), and for infrastructure managed by multiple people. State can therefore also be stored in a distributed filesystem, a distributed config-system such as consul, or in Terraform’s commercial admin manager.

Terraform for non-IaaS environments

As can be seen from the list of providers, Terraform can also be used to automate configuration of things other than IaaS clouds. In fact, anything that offers a management API for creating/deleting resources can be “automated” via Terraform. Interesting examples are:

  • a provider for Docker (local) and Docker-Swarm (distributed); logical as Docker is in some respects similar to an IaaS cloud.
  • a provider for Kubernetes (similar to Docker case)
  • a provider for Heroku (Platform-as-a-Service ie PaaS),
  • providers for Bitbucket/GitHub/GitLab (Git repository hosting) which allows Terraform to define git repositories, webhooks, and assign default reviewers for commits.
  • a provider for Chef Server (OS configuration management) which pushes configuration-data (variable-definitions) from Terraform into Chef, allowing Chef to then use these values when pushing software configuration into VM images.
  • a provider for Cloudflare (Content Distribution Network) to push configuration data into Cloudflare
  • a provider for DNS (Domain Name Service) which allows domain-name records to be pushed to a domain name server.
  • a provider for Graphana, allowing Terraform to configure Graphana “dashboards” in an automated way
  • providers for MySQL and Postgres (SQL database), allowing Terraform to define databases, users and permissions.

Further Reading