Wrong 'allowaccess' order will force Terraform to always commit the changes again

[tkoeck]

Terraform will perform the following actions:

  # fortios_system_interface.internal will be updated in-place
  ~ resource "fortios_system_interface" "internal" {
        id                                         = "internal"
        name                                       = "internal"
        # (165 unchanged attributes hidden)

      + member {
          + interface_name = "internal1 internal internal3 internal4 internal5 internal6"
        }

      ~ secondaryip {
          ~ allowaccess      = "ping https http" -> "ping http https"
            id               = 1
          ~ ip               = "192.168.69.1 255.255.255.0" -> "192.168.69.1/24"
            # (4 unchanged attributes hidden)
        }
      ~ secondaryip {
          ~ allowaccess      = "ping https http" -> "ping http https"
            id               = 2
          ~ ip               = "192.168.1.99 255.255.255.0" -> "192.168.1.99/24"
            # (4 unchanged attributes hidden)
        }

        # (1 unchanged block hidden)
    }

  # fortios_system_interface.vpn will be updated in-place
  ~ resource "fortios_system_interface" "vpn" {
      ~ allowaccess                                = "ping https http" -> "ping http https"
        id                                         = "ipsec_tunnel"
        name                                       = "ipsec_tunnel"
        # (163 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

  # fortios_system_interface.wan1 will be updated in-place
  ~ resource "fortios_system_interface" "wan1" {
      ~ allowaccess                                = "ping https http fgfm" -> "ping fgfm https http"
        id                                         = "wan1"
        name                                       = "wan1"
        # (162 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

Plan: 0 to add, 3 to change, 0 to destroy.

So I can always press yes and the next time it wants to change it, too.

It appears to me that your terraform provider always wants to change that if it is not in the right order

E.g. is not allowaccess = "ping https http fgfm" or for the secondary_ip "ping https http"

[tkoeck]

A similar issue seems to be with the CIDR notation in the seconary_ips block

  secondaryip {
    ip          = "192.168.69.1/24"
    allowaccess = "ping https http"
  }

  secondaryip {
    ip = "192.168.1.99/24"
    allowaccess = "ping https http"
  }

Terraform will always rerun it

Terraform will perform the following actions:

  # fortios_system_interface.internal will be updated in-place
  ~ resource "fortios_system_interface" "internal" {
        id                                         = "internal"
        name                                       = "internal"
        # (165 unchanged attributes hidden)

      + member {
          + interface_name = "internal1 internal internal3 internal4 internal5 internal6"
        }

      ~ secondaryip {
            id               = 1
          ~ ip               = "192.168.69.1 255.255.255.0" -> "192.168.69.1/24"
            # (5 unchanged attributes hidden)
        }
      ~ secondaryip {
            id               = 2
          ~ ip               = "192.168.1.99 255.255.255.0" -> "192.168.1.99/24"
            # (5 unchanged attributes hidden)
        }

        # (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

So it seems to be necessary to write the ip as x.y.z.u a.b.c.d instead of x.y.z.u/n? Even as the documentation says

ip - Interface IPv4 address and subnet mask, syntax: X.X.X.X/24?

[tkoeck]

I might have found the problem. The terraform provider doesn't seem to complain if one interface name is wrong.

[tkoeck]

Actually, it still seems to be the case with the CIDR

locals {
  lan_1 ="192.168.68.254/24"
  lan_2 = "192.168.69.254/24"
  lan_3 = "192.168.1.254/24"
}

resource fortios_system_interface "internal" {
  name                           = "internal"
  type                           = "hard-switch"
  vdom                           = "root"
  mode                           = "static"
  allowaccess                    = "ping https http"
  role                           = "lan"
  ip                             = local.lan_1
  autogenerated = "auto"
  secondary_ip = "enable"

  member {
    interface_name = "internal1 internal2 internal3 internal4 internal5 internal6"
  }

  secondaryip {
    ip          = local.lan_2
    allowaccess = "ping https http"
  }

  secondaryip {
    ip = local.lan_3
    allowaccess = "ping https http"
  }
}
  }

  secondaryip {
    ip = local.lan_3
    allowaccess = "ping https http"
  }

Terraform will perform the following actions:

  # fortios_system_interface.internal will be updated in-place
  ~ resource "fortios_system_interface" "internal" {
        id                                         = "internal"
        name                                       = "internal"
        # (165 unchanged attributes hidden)

      + member {
          + interface_name = "internal1 internal2 internal3 internal4 internal5 internal6"
        }

      ~ secondaryip {
            id               = 1
          ~ ip               = "192.168.69.254 255.255.255.0" -> "192.168.69.254/24"
            # (5 unchanged attributes hidden)
        }
      ~ secondaryip {
            id               = 2
          ~ ip               = "192.168.1.254 255.255.255.0" -> "192.168.1.254/24"
            # (5 unchanged attributes hidden)
        }

        # (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

fortios_system_interface.internal: Modifying... [id=internal]
fortios_system_interface.internal: Modifications complete after 0s [id=internal]

Terraform want's to repeat it every time I apply it.

[tkoeck]

Maybe the reason is

2023-05-16T17:42:40.705+0200 [WARN]  ValidateProviderConfig from "provider[\"registry.terraform.io/fortinetdev/fortios\"]" changed the config value, but that value is unused
2023-05-16T17:42:40.754+0200 [WARN]  Provider "registry.terraform.io/fortinetdev/fortios" produced an invalid plan for fortios_system_dns.dns, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .dynamic_sort_subtable: planned value cty.StringVal("false") for a non-computed attribute
2023-05-16T17:42:40.840+0200 [WARN]  Provider "registry.terraform.io/fortinetdev/fortios" produced an invalid plan for fortios_system_interface.wan1, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .description: planned value cty.StringVal("") for a non-computed attribute
      - .dynamic_sort_subtable: planned value cty.StringVal("false") for a non-computed attribute
      - .ipv6: block count in plan (1) disagrees with count in config (0)
2023-05-16T17:42:40.863+0200 [WARN]  Provider "registry.terraform.io/fortinetdev/fortios" produced an invalid plan for fortios_system_interface.internal, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .description: planned value cty.StringVal("") for a non-computed attribute
      - .dynamic_sort_subtable: planned value cty.StringVal("false") for a non-computed attribute
      - .ipv6: block count in plan (1) disagrees with count in config (0)

[MaxxLiu22]

Hi @tkoeck ,

Thank you for bring up this issue, we will improve this feature in the future, even FOS accepts two types ip/subnet input, but in database, it is stored as one form, so far, if we want to ignore this change msg, we suggest to use default format such as 1.2.3.4 255.255.255.0, sorry for any inconvenience, we will improve the difference comparation feature.

Thanks,
Maxx

[tkoeck]

Hi Maxx,

I adjusted the IP address formats. It still wants to apply and reapply the Terraform configuration.

Terraform will perform the following actions:

  # fortios_router_static.default will be updated in-place
  ~ resource "fortios_router_static" "default" {
        id                    = "1"
      + virtual_wan_link      = "disable"
        # (15 unchanged attributes hidden)
    }

  # fortios_system_interface.internal will be updated in-place
  ~ resource "fortios_system_interface" "internal" {
        id                                         = "internal"
        name                                       = "internal"
        # (165 unchanged attributes hidden)

      + member {
          + interface_name = "internal1 internal2 internal3 internal4 internal5 internal6"
        }

        # (3 unchanged blocks hidden)
    }

Plan: 0 to add, 2 to change, 0 to destroy.

So there seems to be something else wrong.

[tkoeck]

On further studies the entry

member {
interface_name = "internal1 internal2 internal3 internal4 internal5 internal6"
}

in the resource

fortios_system_interface "internal"

doesn't seem to have any effect. Terraform always wants to rerun it.

[MaxxLiu22]

Hi @tkoeck,

Could you try to separate interface info their own member block just like secondaryip

member {
    interface_name = "internal1"
  }
member {
    interface_name = "internal2"
  }
member {
    interface_name = "internal3"
  }

Let me know if that doesn't work.

Thanks,
Maxx

[tkoeck]

Hi Maxx,

yes, I already tried that before. Didn't make a difference.

[MaxxLiu22]

Could you do this operation through GUI? maybe there is any internal interface not available to be added, GUI can help you to check if all of these interfaces can be used by clicking the plus expand icon next to the interface members

[tkoeck]

Yes, I can. I can remove or add all interfaces. I have attached a screenshot of the GUI.

[MaxxLiu22]

I just go through doc, argument member is used by aggregate or redundant interface,, I find there is an argument called "interface", could you try to set interface = "internal1 internal2 internal3 internal4 internal5 internal6" and run again?

[tkoeck]

That doesn't seem to work

tkoeck@tron-nb:~/gitlab/router-berlin$ terraform apply
╷
│ Error: expected length of interface to be in the range (0 - 15), got internal1 internal2 internal3 internal4 internal5 internal6

[mathieubrun]

I was able to work around this plan/apply loop for "wrong" orders by using dynamic_sort_subtable="alphabetical" and making sure the terraform code has alphabetical order as well.
Not ideal at all, but it's doing the job.

[tkoeck]

No change.

resource fortios_system_interface "internal" {
  name                  = "internal"
  type                  = "hard-switch"
  vdom                  = "root"
  mode                  = "static"
  allowaccess           = "ping https ssh http"
  role                  = "lan"
  ip                    = local.lan_1
  #autogenerated = "auto"
  secondary_ip          = "enable"
  dynamic_sort_subtable = "alphabetical"

  secondaryip {
    ip          = local.lan_2
    allowaccess = "ping https http"
  }
  member {
    interface_name = "internal1 internal2 internal3 internal4 internal5 internal6"
  }
}

Terraform will perform the following actions:

  # fortios_system_interface.internal will be updated in-place
  ~ resource "fortios_system_interface" "internal" {
        id                                         = "internal"
        name                                       = "internal"
        # (165 unchanged attributes hidden)

      + member {
          + interface_name = "internal1 internal2 internal3 internal4 internal5 internal6"
        }

        # (2 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: 

[MaxxLiu22]

Hi @tkoeck ,

I find this instruction, according to what here said, adding member interface would add port in fortios_system_virtualswitch resource

Thanks,
Maxx

[zippanto]

Hi @tkoeck ,

Thank you for bring up this issue, we will improve this feature in the future, even FOS accepts two types ip/subnet input, but in database, it is stored as one form, so far, if we want to ignore this change msg, we suggest to use default format such as 1.2.3.4 255.255.255.0, sorry for any inconvenience, we will improve the difference comparation feature.

Thanks, Maxx

Hi @MaxxLiu22,

Just wondering if there is any progress on this issue with having to specify netmask in dotted decimal format? Should we open a separate issue about that?

Thanks.

[MaxxLiu22]

Hi all,

I hope you're doing well. The argument ip compatibility issue should be addressed in Terraform FOS 1.21.1. If possible, please feel free to upgrade and see if this resolves the issue.

Thank you for your patience, and don't hesitate to reach out if you have any questions.

Best regards,
Maxx

[zippanto]

Hi @MaxxLiu22,

That seems to be fixed on interface secondaryip, but it seems the issue is still present on other resources / attributes. Please see below.

  # fortios_router_accesslist.accesslist will be updated in-place
  ~ resource "fortios_router_accesslist" "accesslist" {
        id                    = "test"
        name                  = "test"
        # (3 unchanged attributes hidden)

      ~ rule {
          - id          = 1 -> null
          ~ prefix      = "192.168.0.0 255.255.255.0" -> "192.168.0.0/24"
            # (4 unchanged attributes hidden)
        }
    }

  # fortios_router_ospf.ospf will be updated in-place
  ~ resource "fortios_router_ospf" "ospf" {
        id                                = "RouterOspf"
        # (27 unchanged attributes hidden)

      ~ area {
            id                                             = "0.0.0.1"
            # (11 unchanged attributes hidden)

          ~ range {
                id                = 1
              ~ prefix            = "192.168.0.0 255.255.255.0" -> "192.168.0.0/24"
                # (3 unchanged attributes hidden)
            }

            # (1 unchanged block hidden)
        }

        # (21 unchanged blocks hidden)
    }

Thanks.