How to Deploy Kubernetes Resources with Terraform [Tutorial]

In this tutorial, we will learn how to use Terraform to deploy Kubernetes resources. We will create a Jenkins CI/CD pipeline that uses Docker containers to run the Terraform commands and then integrate it with our minikube cluster. Additionally, we will explore using Spacelift stack dependencies to deploy an app in Kubernetes with Terraform. Prerequisites: - Jenkins installed on your local machine - Minikube installed on your local machine - GitHub account - Docker installed on your local machine Step 1: Create a Dockerfile for the Jenkins image Create a new file named Dockerfile in your project directory with the following content: ``` FROM jenkins/jenkins:lts RUN apt-get update && apt-get install -yq \ curl \ git \ openssh-client \ zip USER root #Install Terraform and kubectl RUN wget https://releases.hashicorp.com/terraform/1.5.3/terraform_1.5.3_linux_amd64.zip && \ unzip terraform_1.5.3_linux_amd64.zip && \ mv terraform /usr/local/bin/ && \ rm terraform_1.5.3_linux_amd64.zip #Install kubectl RUN curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" && \ install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl && \ rm kubectl USER jenkins #Access jenkins on port 8080 EXPOSE 8080 CMD ["jenkins.sh"] ``` Step 2: Build the Docker image for Jenkins Now from the directory you created the Dockerfile in, run the following: ``` #To create the docker image docker build . -t jenkins-image:v1 #Validate image exists docker images ``` Step 3: Run the Jenkins instance We will run the Jenkins instance using this image. To ensure Jenkins can communicate with our minikube instance, we will mount our ~/.kube/config and /Users/your username/.minikube directory to the Jenkins container. Note: The directory path for the minikube certificates will be different if you are using Windows, this will work for MacOS. ``` #Create Jenkins container docker run -d -p 8080:8080 -p 50000:50000 -v ~/.kube:/var/jenkins_home/.kube -v jenkins_home:/var/jenkins_home -v /Users/your-username/.minikube:/Users/yourusername/.minikube --name jenkins-instance jenkins-image:v1 #Validate terraform and kubectl are working on container docker exec jenkins kubectl —-help docker exec jenkins terraform —-help #Grab the password, we will need this to login to Jenkins docker exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword ``` Step 4: Set up the Jenkins instance Follow these steps to set up your Jenkins instance: - Open your web browser and go to http://localhost:8080 to access your Jenkins CI/CD. - Once you are prompted to enter your admin password, enter the password we grabbed from the previous step. - Go through the setup pages to install the recommended plugins and create your admin user, which you will use to log in from then on. - Go to Manage Jenkins > Plugins > Available plugins and install the following plugins: - Kubernetes - Kubernetes CLI - Terraform We should be all set to start creating our Terraform resources and our Jenkins pipeline. Let’s start setting up our Terraform code, which will contain the Kubernetes resources we will deploy to our minikube cluster through Jenkins. You will need to host your Terraform code on a GitHub repository. We will not cover that in this article, but you just need to make sure the repository is public so we can access it from our Jenkins pipeline. In your GitHub repository, create the following main.tf file: ``` provider "kubernetes" { config_path = "~/.kube/config" } #Kubernetes namespace to hold application resource "kubernetes_namespace" "terraform-k8s" { metadata { name = "terraform-k8s" } } resource "kubernetes_deployment" "test-deploy" { metadata { name = "terraform" namespace = "terraform-k8s" labels = { test = "MyApp" } } spec { replicas = 3 selector { match_labels = { test = "MyApp" } } template { metadata { labels = { test = "MyApp" } } spec { container { image = "nginx:1.21.6" name = "nginx-terraform" resources { limits = { cpu = "0.5" memory = "512Mi" } requests = { cpu = "250m" memory = "50Mi" } } liveness_probe { http_get { path = "/" port = 80 http_header { name = "X-Custom-Header" value = "Awesome" } } initial_delay_seconds = 3 period_seconds = 3 } } } } } } ``` Create the Jenkins pipeline We will now return to our Jenkins instance and create the pipeline. For this example, we will add the pipeline solely to deploy the Kubernetes resources using Terraform. However, this can be used alongside the other CI/CD steps in your pipeline. Go to your Jenkins instance and create a new item, select Pipeline, and name it “k8s-terraform-pipeline”. Add the following pipeline script: ``` pipeline{ agent any stages{ stage('Git Checkout'){ steps{ git branch: 'main', url: 'https://github.com/yourusername/repo_name' } } stage('Terraform init'){ steps{ dir("terraform/spacelift/terraform-k8s"){ sh 'terraform init' } } } stage('Terraform plan'){ steps{ dir("terraform/spacelift/terraform-k8s"){ sh 'terraform plan' } } } stage('Terraform apply'){ steps{ dir("terraform/spacelift/terraform-k8s"){ sh 'terraform apply --auto-approve' } } } } } ``` Now save the pipeline and run the build. Once the pipeline runs successfully, go to your terminal and run the following to ensure the application was deployed successfully to your Kubernetes cluster through the Jenkins CI/CD pipeline: ``` kubectl get deployment -n terraform-k8s NAME READY UP-TO-DATE AVAILABLE AGE nginx 1/1 1 1 9m22s kubectl get service -n terraform-k8s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx LoadBalancer 10.x.x.x 172.170.55.37 80:30302/TCP 9m24s ``` Using Spacelift stack dependencies to deploy an app in Kubernetes with Terraform To simplify your CI/CD processes, you can also leverage Spacelift stack dependencies. The stack dependencies feature allows you to define and manage the relationships between different stacks, ensuring that your pipeline steps run in the correct order and all dependencies are resolved before proceeding to specific steps. This is particularly useful in complex infrastructure environments where multiple components — such as account creation and assignments, networking, databases, compute resources, Kubernetes clusters, monitoring, and alerting — must be provisioned in a precise sequence. You can easily pass information between stacks, which not only minimizes the risk of errors but also streamlines the deployment process, making it more efficient and reliable. When using stack dependencies in CI/CD pipelines, you achieve a smoother and more controlled deployment. In this example, we will demonstrate using stack dependencies to deploy Kubernetes resources to an Azure AKS cluster. We will break this process into two stacks: The first stack will focus on deploying an AKS cluster with Terraform, and the second stack will handle deploying Kubernetes resources into the AKS cluster. We will establish a dependency from stack 1 to stack 2, passing the kube config generated by the AKS cluster (stack 1) as an output to the Kubernetes stack (stack 2). Prerequisites - Spacelift account (you can sign up for a 14-day free trial if needed) - Azure subscription - GitHub repo Configure cloud integration In your Spacelift account, go to the Cloud Integrations side tab, click on Azure and Create Integration. Add your Azure Tenant ID and Subscription ID here, and then click Create Integration. You will get a prompt to Provide Consent. Click that and wait a few minutes. Go to your Azure portal, click on Microsoft Entra ID > Enterprise Applications and search for ‘spacelift’. Confirm you can see the application there. If you do not see the application, validate that you have administrator permissions in this Azure subscription and try again. Now, in your Azure Portal, go to your Subscription > IAM > Add Role Assignment. Select the Privileged administrator roles tab and select Contributor. Assign this role to the enterprise application Spacelift. Review the changes and click on Review and assign. Set up Git repositories for Terraform and Kubernetes resources We will create two separate directories in the GitHub repository. One will have the Terraform code to deploy our Azure AKS cluster deployment (we will use the terraform k8s module). The second directory will include the code for our Kubernetes manifest. From the repository’s root directory, create a directory named ‘tf’. Add the following main.tf file: ``` provider "azurerm" { features {} } resource "azurerm_resource_group" "azrg" { name = "az-rg" location = "centralus" } module "aks" { source = "github.com/flavius-dinu/terraform-az-aks.git?ref=v1.0.12" kube_params = { kube1 = { name = "kube1" rg_name = azurerm_resource_group.azrg.name rg_location = "centralus" dns_prefix = "kube" identity = [{}] enable_auto_scaling = false node_count = 1 np_name = "kube1" export_kube_config = false tags = {} } } } output "kube_config" { value = module.aks.kube_config["kube1"] sensitive = true } ``` Again, in the repository’s root directory, create another directory name it ‘k8s’.Then, add the following nginx.yaml file: ``` apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.21.1 ports: - containerPort: 80 ``` Once you have created the files and set them up in your Git repository, you can start working on creating the stacks that will reference these directories. Set up Terraform (Azure AKS Cluster) and Kubernetes stacks with dependencies Terraform stack creation On Spacelift’s home page, click Create Stack. Name your stack “azure-aks-terraform”. If you already logged in using your Github account to Spacelift, you should see your Github repositories. Select your Repository, Branch, and Project Root, which should point to the ‘tf’ directory we created earlier in your Git repository. If you did not log in to Spacelift using your GitHub account, you can log in here, point to your Git repository, and add your project root. On the Choose Vendor screen, make sure Terraform / OpenTofu is selected, and everything else should be set to default. Keep the rest of the screens as default, and click Confirm. Now, we will trigger this stack to create the outputs for our AKS cluster and insert these outputs into the Kubernetes stack dependencies. Go to the azure-aks-terraform stack and click Trigger. This should start creating all your Azure resources via Terraform. After the Terraform Plan is complete, you will be prompted to Confirm the run (Terraform Approve). Make sure to click Confirm. Once the stack is complete, validate that the Azure resources and outputs are created successfully. You can also check the Azure Portal and make sure the resources were created successfully. Kubernetes stack creation Let’s return to Spacelift’s homepage and create the second stack for Kubernetes resource deployment. You can name this one “Kubernetes.” For the Git source code, we will point this to the same repository and branch as the previous stack, but the project root will point to “k8s”’instead. On the Choose vendor screen, select Kubernetes and leave everything else as default. Let’s skip to the Add hooks screen. We will add a “Before Initialization” workflow step to process the outputs from the first stack. We will explain more later, but for now add the following commands to the Before window: ``` mkdir /mnt/workspace/.kube printf "%s\n" "$kubeconfig" > /mnt/workspace/.kube/config ``` Let’s skip to the summary and click Confirm. Go to your ‘azure-aks-terraform’ stack, click Settings, and click Integrations there. Select your Azure subscription here (you should be able to click the drop-down and have it auto-populate the Subscription ID) and make sure to assign it read/write permissions. Output/dependencies In the ‘azure-aks-terraform’ stack, go to the Dependencies tab and under Depended on by, click Add Dependencies. Select Kubernetes and click Add. On the same screen, click Add output reference. Select ‘kube_config’ and enter ‘kubeconfig’ as the Input name. This will insert the Terraform Azure AKS cluster output we placed in our Terraform code into the Kubernetes stack. You can now go to the Spacelift homepage. Under stacks select ‘azure-aks-terraform’ stack and click Trigger. This will run the Terraform stack first (it should not add anything new because we ran this earlier) and place the downstream stack for Kubernetes on “Queued” status. You will need to Confirm the Terraform deployment. Once the stack has finished running, the second stack for Kubernetes will get triggered and create the nginx application on the AKS cluster. You can also validate the Kubernetes deployment was successful on the Azure Portal or through kubectl. And that’s it! You have successfully deployed Kubernetes resources to an Azure AKS cluster via Terraform using Spacelift stack dependencies. If you want to learn more about Spacelift, create a free account today, or book a demo with one of our engineers. Let’s review some best practices for using Terraform to deploy Kubernetes resources: - Modularize your Kubernetes Terraform code | By modularizing your resources, such as Kubernetes deployments, services, config maps, roles, and role bindings, you can simplify deployment and enable reusability. | - Utilize variables | Always try to use variables in your Kubernetes instead of hard coding any values. You can also pass in environmental, regional, and other values from your pipeline down to your Kubernetes modules. | - State management | Use the remote backend (Azure Storage Account or AWS S3 Bucket) to store your remote state file. Also, make sure to use state locking to prevent multiple changes to the terraform state at the same time. | - Outputs | Utilize outputs as much as possible during cluster deployments using Terraform to get kubeconfig and other values, such as cluster endpoint, and use them within your Kubernetes resource deployments. | - IAM role management | Use AWS IAM roles for service accounts (IRSA) or Azure Managed Service Identity (MSI) to manage permissions for your Kubernetes workloads. | - Version control | Make sure to version control your Terraform code and use a CI/CD pipeline to automate the deployment process. | By following these best practices, you can ensure that your Kubernetes resources are deployed consistently and securely using Terraform.