This is a series of Blog posts about Terraform and Azure Devops Services.
Table of Contents
Deploying on Azure with Azure DevOps Services and Terraform
What will you learn from these Blogs?
- Working with Azure DevOps Services, like Azure Repos (like: creating branches, pull requests, code reviews), Azure Pipelines, Azure Artifacts etc.
- Creating the Terraform backend, SPN, Key Vault and secrets in Azure with Azure CLI Powershell script
- Creating a Terraform Multi-Stage YAML pipeline with Manual Validation task, Artifacts, Approvals, Triggers and more
- Integration with Azure Key vault, Terraform and Azure DevOps Pipeline Libraries (Variable groups)
- Working with Visual Studio Code
The result
This will lead to the following work flow, all integrated in the Multi-Stage YAML pipeline:
Each process number in the workflow is described below:
Workflow # | Description |
|---|---|
1. | Engineer creates new branch to amend or create new code |
2. | Code is committed to new branch (locally) |
3. | Engineer pushes the code from local to remote repo (in Azure DevOps) |
4. | Engineer creates a pull request to merge with main branch (from recently created branch in step 1.) |
5. | Pipeline is triggered to validate & plan with Terraform |
6. | The pull request also triggers a review, other engineer(s) are asked (automatically) to review the code |
6a. | When the code is rejected, depending on whether the code can be changed for approval. The pull request will be closed and the code can be changed to restart the process |
7. | After the merge is complete the Terraform Build & Release pipeline is triggered |
8. | Before the pipeline starts you need to approve |
9. | The pipeline runs and Terraform “Plan” stage starts |
10. | The tf.plan file is copied and stored as an artifact to be used in “Apply” stage (step 12.) |
11. | Reviewers check if the Terraform plan will deploy what is expected, if not the release can be rejected, if it is ok we move on to the Terraform “Apply” stage |
12. | Terraform “Apply” stage is initiated |
13. | Resources are deployed and/or destroyed |
Lets get started
In this post you will learn how to create the Terraform backend in Azure with Azure CLI and configure Azure DevOps. For each step I will explain what is built, what is used and why.
Terraform Backend Deployment
What will we deploy?
- Resource Group is a container that holds related resources for an Azure solution ( in this case the Azure Key Vault and Storage account).
- SPN is needed for pipelines (service connection)
- Key Vault is needed to store the secrets for storage account keys, SPN information and credentials.
- Storage account is needed for the Blob container which will host the terraform state file.
Terraform must store state about your managed infrastructure and configuration. This state is used by Terraform to map real world resources to your configuration, keep track of metadata, and to improve performance for large infrastructures.
Deploy script
- Go to the Azure Portal > https://portal.azure.com/ (Free Trial Azure subscription)
- Open ‘Cloud Shell’ (PowerShell)
To view the subscription id of your current login
az account list -o table
If you have multiple subscriptions, select the one you want to use
az account set --subscription "<your subscription id>"
Upload or create file that contains the following script:
Each step in the script is commented to show you what is created.
# Set Variables
$RANDOM = Get-Random -Minimum 100 -Maximum 200
$RESOURCE_GROUP_NAME="rg-terraform-infra-$RANDOM"
$STORAGE_ACCOUNT_NAME="stateterraform$RANDOM"
$CONTAINER_NAME="ct-terraform-state-$RANDOM"
$LOCATION="westeurope"
$SPN_NAME="terraform-azdevops-$RANDOM"
$KEYVAULT="kvterraform$RANDOM"
$OUTPUT="none" # Set to "none" for no output or "json" for default output
# Create resource group
az group create --name $RESOURCE_GROUP_NAME --location $LOCATION --output $OUTPUT
# Create storage account with public access disabled and minimal TLS version
echo "Creating storage account "$STORAGE_ACCOUNT_NAME" in "$RESOURCE_GROUP_NAME"..."
az storage account create --resource-group $RESOURCE_GROUP_NAME --name $STORAGE_ACCOUNT_NAME --sku Standard_LRS --encryption-services blob --min-tls-version TLS1_2 --allow-blob-public-access false --output $OUTPUT
# If storage account gets deleted or corrupt you can restore from the past 7 days
echo "Setting delete retention to 7 days"
az storage account blob-service-properties update --enable-container-delete-retention true --container-delete-retention-days 7 --enable-versioning true --account-name $STORAGE_ACCOUNT_NAME --resource-group $RESOURCE_GROUP_NAME --output $OUTPUT
# Get storage account key
$ACCOUNT_KEY1=$(az storage account keys list --resource-group "$RESOURCE_GROUP_NAME" --account-name "$STORAGE_ACCOUNT_NAME" --query '[0].value' -o tsv)
$ACCOUNT_KEY2=$(az storage account keys list --resource-group "$RESOURCE_GROUP_NAME" --account-name "$STORAGE_ACCOUNT_NAME" --query '[1].value' -o tsv)
# Create blob container
echo "Create blob container"
az storage container create --name $CONTAINER_NAME --account-name $STORAGE_ACCOUNT_NAME --account-key $ACCOUNT_KEY1 --output $OUTPUT
#az storage container create --name $CONTAINER_NAME --account-name $STORAGE_ACCOUNT_NAME --auth-mode login --output $OUTPUT
# Create KeyVault and store keys
echo "Create KeyVault and store keys"
az keyvault create --name "$KEYVAULT" --resource-group "$RESOURCE_GROUP_NAME" --location "$LOCATION" --output $OUTPUT
az keyvault secret set --vault-name "$KEYVAULT" --name "$STORAGE_ACCOUNT_NAME-key1" --value "$ACCOUNT_KEY1" --output $OUTPUT
az keyvault secret set --vault-name "$KEYVAULT" --name "$STORAGE_ACCOUNT_NAME-key2" --value "$ACCOUNT_KEY2" --output $OUTPUT
# Create SPN, add contributor role and store password in keyvault
echo "Create SPN and store password in keyvault"
$SPN_SECRET=$(az ad sp create-for-rbac --name "$SPN_NAME" --role contributor --scopes /subscriptions/$(az account show --query id -o tsv) --query password -o tsv)
$SPN_CLIENT_ID=$(az ad sp list --display-name $SPN_NAME --query [].appId -o tsv)
$SPN_TENANT_ID=$(az ad sp list --display-name $SPN_NAME --query [].appOwnerOrganizationId -o tsv)
az keyvault secret set --vault-name "$KEYVAULT" --name "terraform-azdevops-spn-secret" --value "$SPN_SECRET" --output $OUTPUT
az keyvault secret set --vault-name "$KEYVAULT" --name "terraform-azdevops-spn-client-id" --value "$SPN_CLIENT_ID" --output $OUTPUT
az keyvault secret set --vault-name "$KEYVAULT" --name "terraform-azdevops-spn-tenant-id" --value "$SPN_TENANT_ID" --output $OUTPUT
#Grant SPN permission to keyvault
echo "Grant SPN permission to keyvault"
az keyvault set-policy -n $KEYVAULT --secret-permissions get list --spn $SPN_CLIENT_ID --output $OUTPUT
Adjust the variables to your liking (for testing purposes I use $RANDOM). If you want to create the file run:
nano tfbackend.ps1
CTRL+X to exit, Yes to save the file and hit Enter
Run the script
./tfbackend.ps1
After running the script check the output. Navigate to the resource group that is created to view the resources and everything has the expected outcome. This will be something like this:
Well done! Next steps Azure DevOps configuration!
Azure DevOps configuration
- Go to the Azure DevOps Portal > https://dev.azure.com/ (First 5 users free)
- Go to your Project or create a Project
Create ‘Service Connection’
A Service Connection is required for Azure DevOps Continuous Build and Continuous Release Pipelines to talk to external and remote services and execute tasks.
- From within your project navigate to ‘Project Settings’ (below-left) > ‘Service connections’ and ‘Create service connection’
- Select ‘Azure Resource Manager”
- Select ‘Service principal (manual)’ The SPN is already created by deploy script above and secrets are saved in Key Vault.
You can find the information needed in the ‘Key Vault’ Secrets
- Click ‘Verify and save’
Create variable group
The variable group is used in the pipeline which references to the variable group and uses the secrets as variables.
- From within your project navigate to ‘Pipelines’ > ‘Library’ and add a ‘Variable group’
- Select ‘Terraform Demo’ (name of the ‘Service Connection’)
- Select the Key Vault and authorize
- Select the variables you want to add (all in this case)
- Click save and on to the next steps
It is also possible to create the ‘Service Connection’ from here (only the ids and secrets aren’t saved in Key Vault, so I prefer using the script).
Initialize repository
- Navigate to ‘Repos’ > ‘Files’
- Select Terraform .gitignore template
- Select Add a README
- Initialize the repository
Why use the Terraform .gitignore template? View comments in the code below:
# Local .terraform directories
**/.terraform/*
# .tfstate files
*.tfstate
*.tfstate.*
# Crash log files
crash.log
# Exclude all .tfvars files, which are likely to contain sentitive data, such as
# password, private keys, and other secrets. These should not be part of version
# control as they are data points which are potentially sensitive and subject
# to change depending on the environment.
#
*.tfvars
# Ignore override files as they are usually used to override resources locally and so
# are not checked in
override.tf
override.tf.json
*_override.tf
*_override.tf.json
# Include override files you do wish to add to version control using negated pattern
#
# !example_override.tf
# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
# example: *tfplan*
# Ignore CLI configuration files
.terraformrc
terraform.rc
Setup Visual Studio Code
- Select ‘Clone in VS Code’
- Select the location where you want to store the repo
Installation and configuration
- Download and install Visual Studio Code
- Download and install git for Windows (or Linux;)
- Add these add-ons:
- Azure Account – Visual Studio Marketplace
- Azure CLI Tools – Visual Studio Marketplace
- Node.js (nodejs.org) (is needed when using Azure Cloud shell in Visual Studio Code)
More information about Visual Studio Code usage: Visual Studio Code User Interface
Clone Repository
We are all set to clone the Azure DevOps repository!
From within your project navigate to navigate to ‘Repos’ > ‘Files’ > Click ‘Clone’
All set! Ready to start with the Multi-Stage YAML pipeline which will be in my next post!
