This is a series of Blog posts about Terraform and Azure Devops Services.
Table of Contents
Deploying on Azure with Azure DevOps Services and Terraform
What will you learn from these Blogs?
- Working with Azure DevOps Services, like Azure Repos (like: creating branches, pull requests, code reviews), Azure Pipelines, Azure Artifacts etc.
- Creating the Terraform backend, SPN, Key Vault and secrets in Azure with Azure CLI Powershell script
- Creating a Terraform Multi-Stage YAML pipeline with Manual Validation task, Artifacts, Approvals, Triggers and more
- Integration with Azure Key vault, Terraform and Azure DevOps Pipeline Libraries (Variable groups)
- Working with Visual Studio Code
Each process number in the workflow is described below:
Engineer creates new branch to amend or create new code
Code is committed to new branch (locally)
Engineer pushes the code from local to remote repo (in Azure DevOps)
Engineer creates a pull request to merge with main branch (from recently created branch in step 1.)
Pipeline is triggered to validate & plan with Terraform
The pull request also triggers a review, other engineer(s) are asked (automatically) to review the code
When the code is rejected, depending on whether the code can be changed for approval. The pull request will be closed and the code can be changed to restart the process
After the merge is complete the Terraform Build & Release pipeline is triggered
Before the pipeline starts you need to approve
The pipeline runs and Terraform “Plan” stage starts
The tf.plan file is copied and stored as an artifact to be used in “Apply” stage (step 12.)
Reviewers check if the Terraform plan will deploy what is expected, if not the release can be rejected, if it is ok we move on to the Terraform “Apply” stage
Terraform “Apply” stage is initiated
Resources are deployed and/or destroyed
Lets get started
In this post you will learn how to create the Terraform backend in Azure with Azure CLI and configure Azure DevOps. For each step I will explain what is built, what is used and why.
Terraform Backend Deployment
What will we deploy?
- Resource Group is a container that holds related resources for an Azure solution ( in this case the Azure Key Vault and Storage account).
- SPN is needed for pipelines (service connection)
- Key Vault is needed to store the secrets for storage account keys, SPN information and credentials.
- Storage account is needed for the Blob container which will host the terraform state file.
To view the subscription id of your current login
az account list -o table
If you have multiple subscriptions, select the one you want to use
az account set --subscription " "
Upload or create file that contains the following script:
Each step in the script is commented to show you what is created.
# Set Variables $RANDOM = Get-Random -Minimum 100 -Maximum 200 $RESOURCE_GROUP_NAME="rg-terraform-infra-$RANDOM" $STORAGE_ACCOUNT_NAME="stateterraform$RANDOM" $CONTAINER_NAME="ct-terraform-state-$RANDOM" $LOCATION="westeurope" $SPN_NAME="terraform-azdevops-$RANDOM" $KEYVAULT="kvterraform$RANDOM" $OUTPUT="none" # Set to "none" for no output or "json" for default output # Create resource group az group create --name $RESOURCE_GROUP_NAME --location $LOCATION --output $OUTPUT # Create storage account with public access disabled and minimal TLS version echo "Creating storage account "$STORAGE_ACCOUNT_NAME" in "$RESOURCE_GROUP_NAME"..." az storage account create --resource-group $RESOURCE_GROUP_NAME --name $STORAGE_ACCOUNT_NAME --sku Standard_LRS --encryption-services blob --min-tls-version TLS1_2 --allow-blob-public-access false --output $OUTPUT # If storage account gets deleted or corrupt you can restore from the past 7 days echo "Setting delete retention to 7 days" az storage account blob-service-properties update --enable-container-delete-retention true --container-delete-retention-days 7 --enable-versioning true --account-name $STORAGE_ACCOUNT_NAME --resource-group $RESOURCE_GROUP_NAME --output $OUTPUT # Get storage account key $ACCOUNT_KEY1=$(az storage account keys list --resource-group "$RESOURCE_GROUP_NAME" --account-name "$STORAGE_ACCOUNT_NAME" --query '.value' -o tsv) $ACCOUNT_KEY2=$(az storage account keys list --resource-group "$RESOURCE_GROUP_NAME" --account-name "$STORAGE_ACCOUNT_NAME" --query '.value' -o tsv) # Create blob container echo "Create blob container" az storage container create --name $CONTAINER_NAME --account-name $STORAGE_ACCOUNT_NAME --account-key $ACCOUNT_KEY1 --output $OUTPUT #az storage container create --name $CONTAINER_NAME --account-name $STORAGE_ACCOUNT_NAME --auth-mode login --output $OUTPUT # Create KeyVault and store keys echo "Create KeyVault and store keys" az keyvault create --name "$KEYVAULT" --resource-group "$RESOURCE_GROUP_NAME" --location "$LOCATION" --output $OUTPUT az keyvault secret set --vault-name "$KEYVAULT" --name "$STORAGE_ACCOUNT_NAME-key1" --value "$ACCOUNT_KEY1" --output $OUTPUT az keyvault secret set --vault-name "$KEYVAULT" --name "$STORAGE_ACCOUNT_NAME-key2" --value "$ACCOUNT_KEY2" --output $OUTPUT # Create SPN, add contributor role and store password in keyvault echo "Create SPN and store password in keyvault" $SPN_SECRET=$(az ad sp create-for-rbac --name "$SPN_NAME" --role contributor --scopes /subscriptions/$(az account show --query id -o tsv) --query password -o tsv) $SPN_CLIENT_ID=$(az ad sp list --display-name $SPN_NAME --query .appId -o tsv) $SPN_TENANT_ID=$(az ad sp list --display-name $SPN_NAME --query .appOwnerOrganizationId -o tsv) az keyvault secret set --vault-name "$KEYVAULT" --name "terraform-azdevops-spn-secret" --value "$SPN_SECRET" --output $OUTPUT az keyvault secret set --vault-name "$KEYVAULT" --name "terraform-azdevops-spn-client-id" --value "$SPN_CLIENT_ID" --output $OUTPUT az keyvault secret set --vault-name "$KEYVAULT" --name "terraform-azdevops-spn-tenant-id" --value "$SPN_TENANT_ID" --output $OUTPUT #Grant SPN permission to keyvault echo "Grant SPN permission to keyvault" az keyvault set-policy -n $KEYVAULT --secret-permissions get list --spn $SPN_CLIENT_ID --output $OUTPUT
Adjust the variables to your liking (for testing purposes I use $RANDOM). If you want to create the file run:
CTRL+X to exit, Yes to save the file and hit Enter
Run the script
After running the script check the output. Navigate to the resource group that is created to view the resources and everything has the expected outcome. This will be something like this:
Well done! Next steps Azure DevOps configuration!
Azure DevOps configuration
- Go to the Azure DevOps Portal > https://dev.azure.com/ (First 5 users free)
- Go to your Project or create a Project
Create ‘Service Connection’
- From within your project navigate to ‘Project Settings’ (below-left) > ‘Service connections’ and ‘Create service connection’
- Select ‘Azure Resource Manager”
- Select ‘Service principal (manual)’ The SPN is already created by deploy script above and secrets are saved in Key Vault.
You can find the information needed in the ‘Key Vault’ Secrets
- Click ‘Verify and save’
Create variable group
- From within your project navigate to ‘Pipelines’ > ‘Library’ and add a ‘Variable group’
- Select ‘Terraform Demo’ (name of the ‘Service Connection’)
- Select the Key Vault and authorize
- Select the variables you want to add (all in this case)
- Click save and on to the next steps
It is also possible to create the ‘Service Connection’ from here (only the ids and secrets aren’t saved in Key Vault, so I prefer using the script).
- Navigate to ‘Repos’ > ‘Files’
- Select Terraform .gitignore template
- Select Add a README
- Initialize the repository
Why use the Terraform .gitignore template? View comments in the code below:
# Local .terraform directories **/.terraform/* # .tfstate files *.tfstate *.tfstate.* # Crash log files crash.log # Exclude all .tfvars files, which are likely to contain sentitive data, such as # password, private keys, and other secrets. These should not be part of version # control as they are data points which are potentially sensitive and subject # to change depending on the environment. # *.tfvars # Ignore override files as they are usually used to override resources locally and so # are not checked in override.tf override.tf.json *_override.tf *_override.tf.json # Include override files you do wish to add to version control using negated pattern # # !example_override.tf # Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan # example: *tfplan* # Ignore CLI configuration files .terraformrc terraform.rc
Setup Visual Studio Code
- Select ‘Clone in VS Code’
- Select the location where you want to store the repo
Installation and configuration
- Download and install Visual Studio Code
- Download and install git for Windows (or Linux;)
- Add these add-ons:
We are all set to clone the Azure DevOps repository!
From within your project navigate to navigate to ‘Repos’ > ‘Files’ > Click ‘Clone’