DevOps Galactic Mission: Operation Terraform

*Author: [Hassan Mehmood]

Welcome!

This repository is your gateway to the DevOps Galactic Mission. Here, you will find all the necessary resources and instructions to successfully complete the mission. Follow the steps and details provided below to reproduce my solution and embark on your own interstellar adventure.

I have carefully documented my journey and made sure to address any potential difficulties you may encounter along the way. Rest assured, my guide is designed to help you navigate through the mission smoothly and without any major obstacles.

I hope you find this guide enjoyable and informative. May your mission be a resounding success, and may the force of DevOps be with you!

---

Table of Contents

1. Mission Brief
2. Mission Tools
3. Task 1: Terraform - Establishing the Outpost
4. Task 2: Docker - Building the Space Beacon
5. Task 3: Helm - Deploying the Space Beacon
6. Feedback

---

Mission Brief

Greetings, Space Engineer! Welcome to the DevOps Galactic Mission: Operation Terraform. Your mission is to establish a Kubernetes outpost in your personal AWS Galaxy using Terraform and deploy the crucial "Space Beacon" microservice using Helm.

Objective

Your objective is to deploy a Kubernetes cluster in your personal AWS Galaxy (account), and then deploy a "Space Beacon" microservice using Helm.

---

Mission Tools

1. Terraform: Infrastructure provisioning tool
2. Docker: Containerization platform
3. Helm: Package manager for Kubernetes
4. AWS EKS: Elastic Kubernetes Service
5. AWS VPC: Virtual Private Cloud
6. AWS ECR: Elastic Container Registry
7. GitHub repository: Mission control center

---

Task 1

Terraform - Establishing the Outpost

Description

Your first task is to lay the foundation of our outpost. Write a Terraform script that:

1. Creates a VPC in your AWS Galaxy with a single public subnet.
2. Sets up an EKS (Elastic Kubernetes Service) cluster in the VPC. To aid this task, you may utilize the official EKS Terraform module.
3. Implements the necessary security measures (security groups, IAM roles, etc.) for the EKS cluster.
4. Outputs the outpost coordinates (kubeconfig) to connect to the EKS cluster.

Solution

1. For creating a VPC with a single public subnet, a VPC module is used. VPC Docs. VPC Module Docs.

Challenges and explanations

• When creating a cluster, you need to specify a VPC and at least two subnets in different Availability Zones. Docs.

azs             = ["eu-central-1a", "eu-central-1b"]  # Specify subnets from two different AZs

public_subnets  = ["10.0.0.0/24", "10.0.1.0/24"]      # Specify subnets from two different AZs

• Enable auto-assigning public IP addresses to EC2 instances.

map_public_ip_on_launch = true

2. For creating an EKS cluster within the VPC, the EKS module is used. VPC Docs. VPC Module Docs.

Challenges and explanations

• Enable all the available logs from the control plane for better troubleshooting the applications.

  cluster_enabled_log_types = [
    "audit",
    "api",
    "authenticator",
    "controllerManager",
    "scheduler"
  ]

• My choice of EC2 instance type is t2.micro, as it is part of a free tier.

• I use EKS managed node instead of self-managed, because it simplifies the management and scaling of worker nodes in an EKS cluster, reducing operational overhead and allowing you to focus more on your applications. While in comparison to Fargate, Using AWS EKS-managed node groups provides more control and flexibility over the underlying infrastructure, enabling advanced customization options and allowing direct access to EC2 instances within the EKS cluster.

• My desired size of the EKS managed node group is 2 because 1 node of a type t2.micro is insufficient for system and space-beacon application pods. In this situation, the best practice is to use node autoscaling. Karpenter Docs.

• Note, this k8s cluster setup is not production ready. It is used only for development purposes. To make it production ready, add at least:

  • Three worker nodes from different availability zones.
  • Configured monitoring with Prometheus, Alertmanager, and Grafana.

3. Security groups and IAM roles for security measurements are deployed within the EKS module.

4. In order to output the EKS cluster kubeconfig, eks-kubeconfig module is used. Eks-kubeconfig Module Docs

Challenges and explanations

• Module eks-kubeconfig should run after the completion of the EKS module.

depends_on = [module.eks]

• It is required to secure our kubeconfig output. Kubeconfig is not displayed in the console or stored in the Terraform state file. Either there will be an error.

sensitive = true

Steps

1. Install terraform. The Docs.

2. Create a user group "terraform" in AWS IAM. The Docs.

• As a best practice, AWS recommends attaching policies to a group instead of attaching a managed policy directly to a user. Then, add the user to the appropriate group.*

3. Assign the needed permissions to the user group "terraform".

• Attach AWS-managed policies:

  • CloudWatchLogsFullAccess
  • IAMFullAccess
  • AWSKeyManagementServicePowerUser
  • AmazonVPCFullAccess
  • AmazonEKSClusterPolicy

• Create an inline policy. JSON is here. Replace 599151311607 with your Account ID.

4. Create a user "terraform" in AWS IAM and add the user to an existing "terraform" user group. The Docs.

5. Create an access key for user "terraform". Docs.

6. Clone the mission control center repository.

 git clone https://github.com/ZhangirK/DevOps-Galactic-Mission.git

7. Use your IAM credentials to authenticate the Terraform AWS provider. Set access key Docs.

export AWS_ACCESS_KEY_ID=

export AWS_SECRET_ACCESS_KEY=

8. Navigate to the terraform directory.

cd $PATH/DevOps-Galactic-Mission/terraform  #replace $PATH with the location of the repository 

9. Initialize the terraform directory.

terraform init

10. Create infrastructure.

terraform apply --auto-approve

11. Redirect kubeconfig to the file.

terraform output kubeconfig > kubeconfig

---

Task 2

Docker - Building the Space Beacon

Description

Next, you need to construct our Space Beacon. Develop a Dockerfile for a simple application using the language of your choice. The application should listen on port 80 and respond to HTTP GET requests with "Greetings from the DevOps Squadron!". Once the Docker image is built, push it to a container registry of your choice (e.g., Docker Hub, AWS ECR, Google Container Registry).

Solution

1. I created a simple application in Python without any frameworks.
2. The lightweight Dockerfile is created. Best practices Docs.

Challenges and explanations

• I used a lightweight official docker image with Python. Also, a specific tag is used instead of the latest.
• I added a non-root user for the security measurements. That is the reason I set the application port to 8080. Port 80 requires root priveleges to expose. However, in the next task my k8s service exposes 80 port.
• I did not use a multistage in this case, because it will not affect to the size of an image.
• Additionally, the vulnerability and best practices image scanner is used. Dockle

3. AWS ECR with a private repository is used as a main container registry. It is created via terraform module ecr. AWS ECR Docs. ECR Module Docs.

Steps

1. AWS ECR private repository is already deployed via terraform in Task 1.

2. Install AWS CLI. Installation Docs.

3. Create an access key for your account. It is not recommended to do it for the root user. Make sure your user has the needed permissions. Docs.

4. Configure your profile in AWS CLI. Configuring Docs.

aws configure

AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE

AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Default region name [None]: eu-central-1

Default output format [None]: json

5. Navigate to the directory with Dockerfile.

cd $PATH/DevOps-Galactic-Mission/app  #replace $PATH with the location of the repository

6. Install Docker. Installation guide.

7. Authenticate your Docker client to the Amazon ECR registry to which you intend to push your image.

aws ecr get-login-password --region eu-central-1 | sudo docker login --username AWS --password-stdin your_aws_account_id.dkr.ecr.eu-central-1.amazonaws.com

Do not run the aws command with sudo!

8. Build the image.

sudo docker build -t your_aws_account_id.dkr.ecr.eu-central-1.amazonaws.com/private-force:0.1.0 .

9. Push the image.

sudo docker push your_aws_account_id.dkr.ecr.eu-central-1.amazonaws.com/private-force:0.1.0

---

Task 3

Helm - Deploying the Space Beacon

Description

With the Space Beacon ready, it's time to deploy. Create a Helm chart for the application. This should include:

1. A deployment configuration for the Space Beacon (the Docker image you created in the previous task).
2. A service to transmit our beacon signal.
3. Any additional elements that you consider vital for a production-grade deployment. Explain your choices and their importance

Solution

A custom Helm is created for this task. Helm Docs.

Command to create a custom Helm chart automatically

helm create space-beacon

This chart contains the following templates:

• deployment
• hpa
• ingress
• service
• serviceaccount

I also changed deployment.yaml, service.yaml and values file.

deployment.yaml:

ports:
  - name: http
    containerPort: {{ .Values.service.targetPort }}  #add an ability to customize the port
    protocol: TCP

service.yaml:

ports:
  - port: {{ .Values.service.port }}
    targetPort: {{ .Values.service.targetPort }}  #add an ability to customize the port. The same as deployment port 
    protocol: TCP
    name: http

values file:

image:
  repository: 599151311607.dkr.ecr.eu-central-1.amazonaws.com/private-force  #the needed image repo
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  tag: "0.1.0"

Note, this helm release is not production ready. It is used only for development purposes. To make it production ready, add at least:

• Three pod replicas.
• Configure Pod Distribution Budget.
• Configure PodAntiAffinity.
• Enable ingress with Cloudflare in order to securely expose applications to the internet.
• Enable hpa for pod autoscaling.
• Configure KEDA.
• Configure additional monitoring and alerting with Prometheus, Alertmanager, and Grafana.
• Configure continuous logging with ELK.

Steps

1. Install helm. Installation Docs.

2. Add kubeconfig of your cluster to ~/.kube/config. remove the <<EOT EOT from the file. The kubeconfig is outputted in the previous task.

3. Navigate to the helm directory.

cd $PATH/DevOps-Galactic-Missionn/helm  #replace $PATH with the location of the repository

4. Install a new release to your k8s cluster.

helm upgrade app ./space-beacon --install -n space --create-namespace --set image.repository=<your account id>.dkr.ecr.eu-central-1.amazonaws.com/private-force --set image.tag=0.1.0

5. Check the functionality of an application. Wait until the pod is running.

kubectl port-forward -n space service/app-space-beacon 8080:80 &  #port-forward the service in the background

curl localhost:8080  #curl the app

---