Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Terraform + Ansible Github Actions #1

Merged
merged 63 commits into from
Feb 13, 2024
Merged

Terraform + Ansible Github Actions #1

merged 63 commits into from
Feb 13, 2024

Conversation

hellais
Copy link
Member

@hellais hellais commented Feb 2, 2024

Specifically we have added support for the following:

  • CD of Terraform IaC deployment via Github Actions
  • CD of ansible configuration provisioning via GH Actions
  • Update pending PR with comments stating the outcome of the terraform and ansible deployment tasks
  • Dynamic ansible inventory file generation using terraform as source
  • Provisioning of secrets and host keys dynamically
  • Support for deploying clickhouse to AWS via terraform and configured via ansible
  • Support for doing CD of oonidataapi with AWS ECS using terraform

Copy link

github-actions bot commented Feb 2, 2024

Terraform Run Output 🤖

Format and Style 🖌success

Initialization ⚙️success

Validation 🤖success

Validation Output

$ terraform validate
Success! The configuration is valid.

Plan 📖success

  • Plan: 4 to add, 3 to change, 4 to destroy.
Show Plan

$ terraform plan
Acquiring state lock. This may take a few moments...
module.terraform_state_backend.data.aws_region.current: Reading...
aws_cloudwatch_log_group.app: Refreshing state... [id=tf-ecs-group/app-dataapi]
data.aws_ssm_parameter.ecs_optimized_ami: Reading...
data.aws_availability_zones.available: Reading...
data.aws_ami.debian_ami: Reading...
aws_ecs_cluster.main: Refreshing state... [id=arn:aws:ecs:eu-central-1:082866812839:cluster/ooni-ecs-cluster]
aws_cloudwatch_log_group.ecs: Refreshing state... [id=tf-ecs-group/ecs-agent]
aws_vpc.main: Refreshing state... [id=vpc-08b101405472a0b46]
aws_acm_certificate.dataapi: Refreshing state... [id=arn:aws:acm:eu-central-1:082866812839:certificate/27b7c5e3-ef70-41e8-a6cf-ac0d58a99613]
aws_iam_role.ecs_service: Refreshing state... [id=ooni_ecs_role]
module.terraform_state_backend.data.aws_region.current: Read complete after 0s [id=eu-central-1]
module.terraform_state_backend.aws_s3_bucket.default[0]: Refreshing state... [id=ooni-production-terraform-state]
aws_iam_role.app_instance: Refreshing state... [id=tf-ecs-ooni-instance-role]
module.terraform_state_backend.aws_dynamodb_table.with_server_side_encryption[0]: Refreshing state... [id=ooni-production-terraform-state-lock]
data.aws_ssm_parameter.ecs_optimized_ami: Read complete after 1s [id=/aws/service/ecs/optimized-ami/amazon-linux-2/recommended]
module.terraform_state_backend.data.aws_iam_policy_document.bucket_policy[0]: Reading...
module.terraform_state_backend.data.aws_iam_policy_document.bucket_policy[0]: Read complete after 0s [id=2050088263]
aws_iam_role_policy.ecs_service: Refreshing state... [id=ooni_ecs_role:ooni_ecs_policy]
data.aws_availability_zones.available: Read complete after 1s [id=eu-central-1]
aws_iam_instance_profile.app: Refreshing state... [id=tf-ecs-instprofile]
module.terraform_state_backend.data.aws_iam_policy_document.aggregated_policy[0]: Reading...
module.terraform_state_backend.data.aws_iam_policy_document.aggregated_policy[0]: Read complete after 0s [id=2050088263]
data.aws_ami.debian_ami: Read complete after 1s [id=ami-0e626c31414223120]
aws_iam_role_policy.instance: Refreshing state... [id=tf-ecs-ooni-instance-role:TfEcsOONIInstanceRole]
aws_ecs_task_definition.dataapi: Refreshing state... [id=ooni-dataapi-production-td]
aws_route53_record.dataapi_cert_validation["dataapi.prod.ooni.io"]: Refreshing state... [id=Z02418652BOD91LFA5S9X__74a0d0e91f58697b6838bed5a2fb4939.dataapi.prod.ooni.io._CNAME]
aws_acm_certificate_validation.dataapi: Refreshing state... [id=2024-02-05 11:11:01.598 +0000 UTC]
aws_internet_gateway.gw: Refreshing state... [id=igw-05319878d828e0a53]
aws_subnet.main[0]: Refreshing state... [id=subnet-06d47d18b015109d4]
aws_alb_target_group.dataapi: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:082866812839:targetgroup/ooni-ecs-dataapi/c16001d24f858f20]
aws_security_group.clickhouse_sg: Refreshing state... [id=sg-0c0ffdfe535bededa]
aws_security_group.lb_sg: Refreshing state... [id=sg-090ff83a71e7ad235]
aws_subnet.main[1]: Refreshing state... [id=subnet-02241f60bd951358d]
aws_route_table.r: Refreshing state... [id=rtb-06814accd93aca9ce]
aws_security_group.instance_sg: Refreshing state... [id=sg-0f39450a61732d69b]
aws_alb.main: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:082866812839:loadbalancer/app/ooni-alb-ecs/4cf8bd00b53f8961]
aws_instance.clickhouse_server_prod_tier1: Refreshing state... [id=i-0eeb82941c837962a]
aws_route_table_association.a[1]: Refreshing state... [id=rtbassoc-034c4831c1754e284]
aws_route_table_association.a[0]: Refreshing state... [id=rtbassoc-049f8c5e84cb32e55]
aws_launch_template.app: Refreshing state... [id=lt-01b923563af0432b5]
module.terraform_state_backend.aws_s3_bucket_versioning.default[0]: Refreshing state... [id=ooni-production-terraform-state]
module.terraform_state_backend.aws_s3_bucket_public_access_block.default[0]: Refreshing state... [id=ooni-production-terraform-state]
module.terraform_state_backend.aws_s3_bucket_server_side_encryption_configuration.default[0]: Refreshing state... [id=ooni-production-terraform-state]
module.terraform_state_backend.aws_s3_bucket_policy.default[0]: Refreshing state... [id=ooni-production-terraform-state]
aws_autoscaling_group.app: Refreshing state... [id=ooni-tier1-production-backend-asg20240119184843559900000002]
module.terraform_state_backend.time_sleep.wait_for_aws_s3_bucket_settings[0]: Refreshing state... [id=2024-02-02T21:32:23Z]
aws_route53_record.alb_dns: Refreshing state... [id=Z02418652BOD91LFA5S9X_dataapi.prod.ooni.io_A]
module.terraform_state_backend.aws_s3_bucket_ownership_controls.default[0]: Refreshing state... [id=ooni-production-terraform-state]
aws_alb_listener.front_end: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:082866812839:listener/app/ooni-alb-ecs/4cf8bd00b53f8961/1cd9240a34a13129]
aws_alb_listener.front_end_https: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:082866812839:listener/app/ooni-alb-ecs/4cf8bd00b53f8961/b7ec77ceadac94c8]
aws_ecs_service.dataapi: Refreshing state... [id=arn:aws:ecs:eu-central-1:082866812839:service/ooni-ecs-cluster/ooni-ecs-dataapi-production]
aws_eip.clickhouse_ip: Refreshing state... [id=eipalloc-0f0a20bba4a1d858c]
aws_ebs_volume.clickhouse_data_volume: Refreshing state... [id=vol-0753909d310ed6d8d]
aws_route53_record.clickhouse_dns: Refreshing state... [id=Z035992527R8VEIX2UVO0_clickhouse.tier1.prod.ooni.nu_A]
aws_volume_attachment.clickhouse_data_volume_attachment: Refreshing state... [id=vai-1098690442]
local_file.ansible_inventory: Refreshing state... [id=eb6f3550caaaa26abcd498ff7f362de78702bd71]
null_resource.ansible_update_known_hosts: Refreshing state... [id=4418919173081010018]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # aws_ebs_volume.clickhouse_data_volume must be replaced
-/+ resource "aws_ebs_volume" "clickhouse_data_volume" {
      ~ arn                  = "arn:aws:ec2:eu-central-1:082866812839:volume/vol-0753909d310ed6d8d" -> (known after apply)
      ~ availability_zone    = "eu-central-1a" # forces replacement -> (known after apply) # forces replacement
      ~ encrypted            = false -> (known after apply)
      ~ id                   = "vol-0753909d310ed6d8d" -> (known after apply)
      ~ iops                 = 3000 -> (known after apply)
      + kms_key_id           = (known after apply)
      - multi_attach_enabled = false -> null
      + snapshot_id          = (known after apply)
        tags                 = {
            "Name"       = "ooni-tier1-production"
            "Repository" = "https://github.com/ooni/devops"
        }
      ~ throughput           = 125 -> (known after apply)
        # (4 unchanged attributes hidden)
    }

  # aws_ecs_service.dataapi will be updated in-place
  ~ resource "aws_ecs_service" "dataapi" {
        id                                 = "arn:aws:ecs:eu-central-1:082866812839:service/ooni-ecs-cluster/ooni-ecs-dataapi-production"
        name                               = "ooni-ecs-dataapi-production"
        tags                               = {
            "Name"       = "ooni-tier1-production"
            "Repository" = "https://github.com/ooni/devops"
        }
      ~ triggers                           = {
          ~ "redeployment" = "2024-02-07T12:21:53Z" -> "2024-02-13T11:00:47Z"
        }
        # (15 unchanged attributes hidden)

        # (3 unchanged blocks hidden)
    }

  # aws_eip.clickhouse_ip will be updated in-place
  ~ resource "aws_eip" "clickhouse_ip" {
        id                   = "eipalloc-0f0a20bba4a1d858c"
      ~ instance             = "i-0eeb82941c837962a" -> (known after apply)
        tags                 = {
            "Name"       = "ooni-tier1-production"
            "Repository" = "https://github.com/ooni/devops"
        }
        # (12 unchanged attributes hidden)
    }

  # aws_instance.clickhouse_server_prod_tier1 must be replaced
-/+ resource "aws_instance" "clickhouse_server_prod_tier1" {
      ~ ami                                  = "ami-04d481f6da046c1e2" -> "ami-0e626c31414223120" # forces replacement
      ~ arn                                  = "arn:aws:ec2:eu-central-1:082866812839:instance/i-0eeb82941c837962a" -> (known after apply)
      ~ availability_zone                    = "eu-central-1a" -> (known after apply)
      ~ cpu_core_count                       = 2 -> (known after apply)
      ~ cpu_threads_per_core                 = 2 -> (known after apply)
      ~ disable_api_stop                     = false -> (known after apply)
      ~ disable_api_termination              = false -> (known after apply)
      ~ ebs_optimized                        = false -> (known after apply)
      - hibernation                          = false -> null
      + host_id                              = (known after apply)
      + host_resource_group_arn              = (known after apply)
      + iam_instance_profile                 = (known after apply)
      ~ id                                   = "i-0eeb82941c837962a" -> (known after apply)
      ~ instance_initiated_shutdown_behavior = "stop" -> (known after apply)
      + instance_lifecycle                   = (known after apply)
      ~ instance_state                       = "running" -> (known after apply)
      ~ ipv6_address_count                   = 0 -> (known after apply)
      ~ ipv6_addresses                       = [] -> (known after apply)
      ~ monitoring                           = false -> (known after apply)
      + outpost_arn                          = (known after apply)
      + password_data                        = (known after apply)
      + placement_group                      = (known after apply)
      ~ placement_partition_number           = 0 -> (known after apply)
      ~ primary_network_interface_id         = "eni-0d22bca3902589e09" -> (known after apply)
      ~ private_dns                          = "ip-10-0-0-39.eu-central-1.compute.internal" -> (known after apply)
      ~ private_ip                           = "10.0.0.39" -> (known after apply)
      + public_dns                           = (known after apply)
      ~ public_ip                            = "3.65.139.104" -> (known after apply)
      ~ secondary_private_ips                = [] -> (known after apply)
      ~ security_groups                      = [] -> (known after apply)
      + spot_instance_request_id             = (known after apply)
        tags                                 = {
            "Name"       = "clickhouse-ooni-tier1-production"
            "Repository" = "https://github.com/ooni/devops"
        }
      ~ tenancy                              = "default" -> (known after apply)
      ~ user_data                            = (sensitive value)
      + user_data_base64                     = (known after apply)
        # (9 unchanged attributes hidden)

      - capacity_reservation_specification {
          - capacity_reservation_preference = "open" -> null
        }

      - cpu_options {
          - core_count       = 2 -> null
          - threads_per_core = 2 -> null
        }

      - ebs_block_device {
          - delete_on_termination = false -> null
          - device_name           = "/dev/sdf" -> null
          - encrypted             = false -> null
          - iops                  = 3000 -> null
          - tags                  = {
              - "Name"       = "ooni-tier1-production"
              - "Repository" = "https://github.com/ooni/devops"
            } -> null
          - throughput            = 125 -> null
          - volume_id             = "vol-0753909d310ed6d8d" -> null
          - volume_size           = 1024 -> null
          - volume_type           = "gp3" -> null
        }

      - enclave_options {
          - enabled = false -> null
        }

      - maintenance_options {
          - auto_recovery = "default" -> null
        }

      - metadata_options {
          - http_endpoint               = "enabled" -> null
          - http_protocol_ipv6          = "disabled" -> null
          - http_put_response_hop_limit = 1 -> null
          - http_tokens                 = "optional" -> null
          - instance_metadata_tags      = "disabled" -> null
        }

      - private_dns_name_options {
          - enable_resource_name_dns_a_record    = false -> null
          - enable_resource_name_dns_aaaa_record = false -> null
          - hostname_type                        = "ip-name" -> null
        }

      ~ root_block_device {
          ~ device_name           = "/dev/xvda" -> (known after apply)
          ~ encrypted             = false -> (known after apply)
          ~ iops                  = 3000 -> (known after apply)
          + kms_key_id            = (known after apply)
          - tags                  = {} -> null
          ~ throughput            = 125 -> (known after apply)
          ~ volume_id             = "vol-00cf114c743e13e50" -> (known after apply)
            # (3 unchanged attributes hidden)
        }
    }

  # aws_launch_template.app will be updated in-place
  ~ resource "aws_launch_template" "app" {
      ~ default_version                      = 8 -> (known after apply)
        id                                   = "lt-01b923563af0432b5"
      ~ image_id                             = (sensitive value)
      ~ latest_version                       = 8 -> (known after apply)
        name                                 = "ooni-tier1-production-backend-lt20240120072522124000000001"
        tags                                 = {}
      ~ user_data                            = (sensitive value)
        # (11 unchanged attributes hidden)

        # (4 unchanged blocks hidden)
    }

  # aws_volume_attachment.clickhouse_data_volume_attachment must be replaced
-/+ resource "aws_volume_attachment" "clickhouse_data_volume_attachment" {
      ~ id           = "vai-1098690442" -> (known after apply)
      ~ instance_id  = "i-0eeb82941c837962a" # forces replacement -> (known after apply) # forces replacement
      ~ volume_id    = "vol-0753909d310ed6d8d" # forces replacement -> (known after apply) # forces replacement
        # (2 unchanged attributes hidden)
    }

  # local_file.ansible_inventory must be replaced
-/+ resource "local_file" "ansible_inventory" {
      ~ content              = <<-EOT # forces replacement
            # Do not edit!
            # autogenerated by terraform in ooni/devops
            [all]
            clickhouse.tier1.prod.ooni.nu
            
            [clickhouse_servers]
            clickhouse.tier1.prod.ooni.nu
        EOT
      ~ content_base64sha256 = "xMMxBEvKeeiPtagiz3q71bbwjrWdvM63a7UUYPFZZzY=" -> (known after apply)
      ~ content_base64sha512 = "r1t3K+ZpFzhhP1Dxhdx3xevYdTW5m8KK/Ema617lq8Sbma1/4P97SIY5vZyNCPinwP7gDZuKBPk8uxGQ2oSItw==" -> (known after apply)
      ~ content_md5          = "7a3a10573536c71358fad39aba48a1ab" -> (known after apply)
      ~ content_sha1         = "eb6f3550caaaa26abcd498ff7f362de78702bd71" -> (known after apply)
      ~ content_sha256       = "c4c331044bca79e88fb5a822cf7abbd5b6f08eb59dbcceb76bb51460f1596736" -> (known after apply)
      ~ content_sha512       = "af5b772be6691738613f50f185dc77c5ebd87535b99bc28afc499aeb5ee5abc49b99ad7fe0ff7b488639bd9c8d08f8a7c0fee00d9b8a04f93cbb1190da8488b7" -> (known after apply)
      ~ id                   = "eb6f3550caaaa26abcd498ff7f362de78702bd71" -> (known after apply)
        # (3 unchanged attributes hidden)
    }

Plan: 4 to add, 3 to change, 4 to destroy.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

Apply 📖``

  • undefined
  • undefined
Show Apply


Pusher @hellais
Action pull_request
Working Directory ./tf/environments/production
Workflow .github/workflows/check_deploy.yml
Last updated Tue, 13 Feb 2024 11:00:52 GMT

@ooni ooni deleted a comment from github-actions bot Feb 2, 2024
@ooni ooni deleted a comment from github-actions bot Feb 2, 2024
@hellais hellais changed the title Terraform github actions Terraform + Ansible github actions Feb 4, 2024
@hellais hellais changed the title Terraform + Ansible github actions Terraform + Ansible Github Actions Feb 4, 2024
Copy link

github-actions bot commented Feb 4, 2024

Ansible Run Output 🤖

Ansible Playbook Recap 🔍


clickhouse.tier1.prod.ooni.nu : ok=9    changed=3    unreachable=0    failed=0    skipped=3    rescued=0    ignored=0

Ansible playbook output 📖success

Show Execution

$ ansible-playbook playbook.yml --check --diff -i inventory.ini

PLAY [ClickHouse servers] ******************************************************

TASK [Gathering Facts] *********************************************************
ok: [clickhouse.tier1.prod.ooni.nu]

TASK [clickhouse : install clickhouse requirements] ****************************
ok: [clickhouse.tier1.prod.ooni.nu]

TASK [clickhouse : Check if ClickHouse GPG keyring exists] *********************
ok: [clickhouse.tier1.prod.ooni.nu]

TASK [clickhouse : Create a temporary directory for GPG] ***********************
skipping: [clickhouse.tier1.prod.ooni.nu]

TASK [clickhouse : Import ClickHouse GPG key] **********************************
skipping: [clickhouse.tier1.prod.ooni.nu]

TASK [clickhouse : Remove temporary directory] *********************************
skipping: [clickhouse.tier1.prod.ooni.nu]

TASK [clickhouse : Ensure the keyring is readable] *****************************
ok: [clickhouse.tier1.prod.ooni.nu]

TASK [clickhouse : Add ClickHouse repository] **********************************
ok: [clickhouse.tier1.prod.ooni.nu]

TASK [clickhouse : Install ClickHouse server and client] ***********************
Suggested packages:
  clickhouse-common-static-dbg
The following packages will be upgraded:
  clickhouse-client clickhouse-common-static clickhouse-server
3 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
changed: [clickhouse.tier1.prod.ooni.nu]

TASK [clickhouse : Ensure ClickHouse service is started and enabled] ***********
ok: [clickhouse.tier1.prod.ooni.nu]

TASK [clickhouse : Configure ClickHouse users from template] *******************
--- before: /etc/clickhouse-server/users.d/ooni_users.xml
+++ after: /home/runner/.ansible/tmp/ansible-local-3782u3x41ime/tmp55y8rluy/ooni_users.xml
@@ -26,4 +26,5 @@
     </reader>
 
   </users>
-</clickhouse>
 No newline at end of file
+</clickhouse>
+

changed: [clickhouse.tier1.prod.ooni.nu]

RUNNING HANDLER [restart clickhouse-server] ************************************
changed: [clickhouse.tier1.prod.ooni.nu]

PLAY RECAP *********************************************************************
clickhouse.tier1.prod.ooni.nu : ok=9    changed=3    unreachable=0    failed=0    skipped=3    rescued=0    ignored=0   

Pusher @hellais
Action pull_request
Working Directory ./tf/environments/production
Workflow .github/workflows/check_deploy.yml
Last updated Tue, 13 Feb 2024 11:03:38 GMT

hellais and others added 18 commits February 6, 2024 12:32
Disable on trigger

Re-enable

Add runs-on

s3 tf backend region should not be a variable

Fix naming of variable
When you try to do it out of the box, you get this error:

```
Initializing the backend...
Migrating from Terraform Cloud to s3 state.
╷
│ Error: Migrating state from Terraform Cloud to another backend is not yet implemented.
│
│ Please use the API to do this: https://www.terraform.io/docs/cloud/api/state-versions.html
```
The instructions on that page are not every helpful.

Luckily this fine fellow on the internet tells us how to do it here:
https://nedinthecloud.com/2022/03/03/migrating-state-data-off-terraform-cloud/

tl;dr is to do this:
1. `terraform init` with the cloud provider
2. `mkdir -p terraform.tfstate.d/tfc-migration-test`
3. `terraform state pull > terraform.tfstate.d/production-tier1/terraform.tfstate`
4. `mv .terraform/terraform.tfstate .terraform/terraform.tfstate.old`
5. Drop the cloud section of the terraform provider and replace it with
   s3
6. `terraform init`
Use terraform_state_backend module to create backend

Add docs on how to use this

Comment out event_name filter

Fix commenting action

hotifx

Update to new github output pattern

Run terraform fmt

Update the last comment instead of creating a new one

Fix GITHUB multiline output

Also, tables are nice

Fix editing of comment?

Fix bugs
Fix push command

ghetto hax
Fix path to ansible inventory file

Fix working directory

fix typo
@ooni ooni deleted a comment from github-actions bot Feb 6, 2024
Copy link
Contributor

@bassosimone bassosimone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM if provided suggestions are applied

.github/workflows/check_deploy.yml Show resolved Hide resolved
.github/workflows/check_deploy.yml Outdated Show resolved Hide resolved
.github/workflows/check_deploy.yml Outdated Show resolved Hide resolved
.github/workflows/check_deploy.yml Outdated Show resolved Hide resolved
scripts/dump-tables-ch.sh Show resolved Hide resolved
tf/environments/production/scripts/update_known_hosts.sh Outdated Show resolved Hide resolved
tf/environments/production/templates/ansible-inventory.tpl Outdated Show resolved Hide resolved
tf/environments/production/templates/clickhouse-setup.sh Outdated Show resolved Hide resolved
tf/environments/production/templates/task_definition.json Outdated Show resolved Hide resolved
Copy link
Contributor

@bassosimone bassosimone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🐳

@hellais hellais added this pull request to the merge queue Feb 13, 2024
Merged via the queue into main with commit a9d6924 Feb 13, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants