Configure aws terraform to allow multiple efs

[GeorgianaElena]

Fixes #4461

[GeorgianaElena]

Terraform output

Terraform planned the following actions, but then encountered a problem:

  # aws_efs_file_system.homedirs will be destroyed
  # (because resource uses count or for_each)
  - resource "aws_efs_file_system" "homedirs" {
      - arn                             = "arn:aws:elasticfilesystem:us-west-2:783616723547:file-system/fs-b25253b5" -> null
      - creation_token                  = "ded6e536cc9e07f14207cd7df6c6f666" -> null
      - dns_name                        = "fs-b25253b5.efs.us-west-2.amazonaws.com" -> null
      - encrypted                       = true -> null
      - id                              = "fs-b25253b5" -> null
      - kms_key_id                      = "arn:aws:kms:us-west-2:783616723547:key/45bb02b5-4366-4f64-a500-7bcf57c4470f" -> null
      - name                            = "hub-homedirs" -> null
      - number_of_mount_targets         = 3 -> null
      - owner_id                        = "783616723547" -> null
      - performance_mode                = "generalPurpose" -> null
      - provisioned_throughput_in_mibps = 0 -> null
      - size_in_bytes                   = [
          - {
              - value             = 1579278168064
              - value_in_ia       = 957880299520
              - value_in_standard = 621397868544
            },
        ] -> null
      - tags                            = {
          - "Name" = "hub-homedirs"
        } -> null
      - tags_all                        = {
          - "Name" = "hub-homedirs"
        } -> null
      - throughput_mode                 = "bursting" -> null
        # (2 unchanged attributes hidden)

      - lifecycle_policy {
          - transition_to_primary_storage_class = "AFTER_1_ACCESS" -> null
            # (2 unchanged attributes hidden)
        }
      - lifecycle_policy {
          - transition_to_ia                    = "AFTER_90_DAYS" -> null
            # (2 unchanged attributes hidden)
        }

      - protection {
          - replication_overwrite = "ENABLED" -> null
        }
    }

  # aws_efs_file_system.homedirs["filestore"] will be created
  + resource "aws_efs_file_system" "homedirs" {
      + arn                     = (known after apply)
      + availability_zone_id    = (known after apply)
      + availability_zone_name  = (known after apply)
      + creation_token          = (known after apply)
      + dns_name                = (known after apply)
      + encrypted               = (known after apply)
      + id                      = (known after apply)
      + kms_key_id              = (known after apply)
      + name                    = (known after apply)
      + number_of_mount_targets = (known after apply)
      + owner_id                = (known after apply)
      + performance_mode        = (known after apply)
      + size_in_bytes           = (known after apply)
      + tags                    = {
          + "Name" = "hub-homedirs"
        }
      + tags_all                = {
          + "Name" = "hub-homedirs"
        }
      + throughput_mode         = "bursting"

      + lifecycle_policy {
          + transition_to_primary_storage_class = "AFTER_1_ACCESS"
        }
      + lifecycle_policy {
          + transition_to_ia = "AFTER_90_DAYS"
        }

      + protection (known after apply)
    }

Plan: 1 to add, 0 to change, 1 to destroy.
╷
│ Error: Instance cannot be destroyed
│ 
│   on efs.tf line 42:
│   42: resource "aws_efs_file_system" "homedirs" {
│ 
│ Resource aws_efs_file_system.homedirs has lifecycle.prevent_destroy set, but the plan calls for this resource to be destroyed.
│ To avoid this error and continue with the plan, either disable lifecycle.prevent_destroy or reduce the scope of the plan using
│ the -target option.

[GeorgianaElena]

Will try a dry-run of terraform state mv as @sgibson91 described in #4373

[GeorgianaElena]

Because of the extra complexities of creating mount targets in each network subnet, there had to be 5 state mv commands:

terraform state mv 'aws_efs_file_system.homedirs' 'aws_efs_file_system.homedirs["filestore"]' 
terraform state mv 'aws_efs_backup_policy.homedirs' 'aws_efs_backup_policy.homedirs["filestore"]'


terraform state mv 'aws_efs_mount_target.homedirs["subnet-0048ebd5f1c78fc2f"]' 'aws_efs_mount_target.homedirs["subnet-0048ebd5f1c78fc2f.fs-b25253b5"]'
terraform state mv 'aws_efs_mount_target.homedirs["subnet-0c6a10d3839a0697d"]' 'aws_efs_mount_target.homedirs["subnet-0c6a10d3839a0697d.fs-b25253b5"]'
terraform state mv 'aws_efs_mount_target.homedirs["subnet-0dcca381809b80b79"]' 'aws_efs_mount_target.homedirs["subnet-0dcca381809b80b79.fs-b25253b5"]'

And now the terraform plan output is clean:

Changes to Outputs:
  ~ nfs_server_dns            = "fs-b25253b5.efs.us-west-2.amazonaws.com" -> [
      + "fs-b25253b5.efs.us-west-2.amazonaws.com",
    ]

You can apply this plan to save these new output values to the Terraform state, without changing any real infrastructure.

[GeorgianaElena]

So, I did something else in order to avoid running the terraform state mv 5 times for all of the aws clusters as I was afraid that process will be very error prone.

The last commit keeps the original EFS related terraform resources and just adds new ones to account for the new map of EFSes that's now possible.

I verified this on the 2i2c-aws-us cluster and with the current code it's a no-op, so that's good.

I will leave this as it is atm because it's late and don't want to mess anything, but tomorrow I will need to run terraform state mv back for the openscapes cluster, and then create a new efs for the staging hub there, and merge this PR once that's done.

So until tomorow terraform state for openscapes is currently broken.

[GeorgianaElena]

In its current state, the terraform plan output now is:

Terraform will perform the following actions:

  # aws_efs_backup_policy.homedirs has moved to aws_efs_backup_policy.homedirs[0]
    resource "aws_efs_backup_policy" "homedirs" {
        id             = "fs-b25253b5"
        # (1 unchanged attribute hidden)

        # (1 unchanged block hidden)
    }

  # aws_efs_file_system.homedirs has moved to aws_efs_file_system.homedirs[0]
    resource "aws_efs_file_system" "homedirs" {
        id                              = "fs-b25253b5"
        name                            = "hub-homedirs"
        tags                            = {
            "Name" = "hub-homedirs"
        }
        # (14 unchanged attributes hidden)

        # (3 unchanged blocks hidden)
    }

Plan: 0 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + nfs_server_dns_list       = []

You can apply this plan to save these new output values to the Terraform state, without changing any real infrastructure.

So I'll go ahead and merge this PR since it's not breaking anything existing