[BUG] index-patterns were created in wrong tenant

[mmuel42]

I tried to create multiple index-patterns which were assigned to different tenants. While applying the known error

│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to module.opensearch.opensearch_dashboard_object.tenant-2-index-1, provider "provider[\"registry.opentofu.org/opensearch-project/opensearch\"]" produced an unexpected new value: Root object was present,
│ but now absent.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

was thrown. After applying again and checking the index-patterns created on the opensearch-cluster, I noticed that some index-patterns were added to the wrong tenant.

Here is an example how I was able to reproduce it quite consistenly. (Sometimes multiple retries were necessary):

How can one reproduce the bug?

using version 2.3.1 of the opensearch-terraform provider and applying the following terraform config:

locals {
  index-patterns = {
    tenant-1-1 = "tenant-1-1"
    tenant-1-2 = "tenant-1-2"
    tenant-1-3 = "tenant-1-3"
    tenant-2-1 = "tenant-2-1"
    tenant-2-2 = "tenant-2-2"
    tenant-2-3 = "tenant-2-3"
  }
  tenants = {
    tenant-1 = {
      name           = "tenant-1"
      description    = "tenant-1"
      index_patterns = [
        local.index-patterns.tenant-1-1,
        local.index-patterns.tenant-1-2,
        local.index-patterns.tenant-1-3
      ]
    }
    tenant-2 = {
      name           = "tenant-2"
      description    = "tenant-2"
      index_patterns = [
        local.index-patterns.tenant-2-1,
        local.index-patterns.tenant-2-2,
        local.index-patterns.tenant-2-3
      ]
    }
  }
}

resource "opensearch_dashboard_tenant" "this" {
  for_each    = local.tenants
  tenant_name = each.value.name
  description = each.value.description

}
resource "opensearch_dashboard_object" "tenant-1-index-patterns" {
  for_each    = toset(local.tenants.tenant-1.index_patterns)
  tenant_name = local.tenants.tenant-1.name
  body = <<EOF
[
  {
    "_id": "index-pattern:${each.key}",
    "_source": {
      "type": "index-pattern",
      "index-pattern": {
        "title": "${each.key}",
        "timeFieldName": "eventTimestamp"
      }
    }
  }
]
EOF
  depends_on = [opensearch_dashboard_tenant.this]
}

resource "opensearch_dashboard_object" "tenant-2-index-patterns" {
  for_each = toset(local.tenants.tenant-2.index_patterns)
  tenant_name = local.tenants.tenant-2.name
  body        = <<EOF
[
  {
    "_id": "index-pattern:${each.key}",
    "_source": {
      "type": "index-pattern",
      "index-pattern": {
        "title": "${each.key}",
        "timeFieldName": "eventTimestamp"
      }
    }
  }
]
EOF
  depends_on = [opensearch_dashboard_tenant.this]
}

What should the provided terraform-configuration do:

• create 2 tenants (tenant-1, tenant-2)
• each tenant has 3 index-patterns assigned:
• tenant-1 should have the index-patterns tenant-1-1, tenant-1-2, tenant-1-3
• tenant-2 should have the index-patterns tenant-2-1, tenant-2-2, `tenant-2-3

after first apply, I'm getting the provider error:

Plan: 8 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  OpenTofu will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.opensearch.opensearch_dashboard_tenant.this["tenant-1"]: Creating...
module.opensearch.opensearch_dashboard_tenant.this["tenant-2"]: Creating...
module.opensearch.opensearch_dashboard_tenant.this["tenant-1"]: Creation complete after 0s [id=tenant-1]
module.opensearch.opensearch_dashboard_tenant.this["tenant-2"]: Creation complete after 1s [id=tenant-2]
module.opensearch.opensearch_dashboard_object.tenant-2-index-patterns["tenant-2-2"]: Creating...
module.opensearch.opensearch_dashboard_object.tenant-2-index-patterns["tenant-2-3"]: Creating...
module.opensearch.opensearch_dashboard_object.tenant-1-index-patterns["tenant-1-2"]: Creating...
module.opensearch.opensearch_dashboard_object.tenant-1-index-patterns["tenant-1-1"]: Creating...
module.opensearch.opensearch_dashboard_object.tenant-2-index-patterns["tenant-2-1"]: Creating...
module.opensearch.opensearch_dashboard_object.tenant-1-index-patterns["tenant-1-3"]: Creating...
module.opensearch.opensearch_dashboard_object.tenant-2-index-patterns["tenant-2-1"]: Creation complete after 0s [id=index-pattern:tenant-2-1]
module.opensearch.opensearch_dashboard_object.tenant-1-index-patterns["tenant-1-2"]: Creation complete after 0s [id=index-pattern:tenant-1-2]
module.opensearch.opensearch_dashboard_object.tenant-1-index-patterns["tenant-1-1"]: Creation complete after 0s [id=index-pattern:tenant-1-1]
module.opensearch.opensearch_dashboard_object.tenant-1-index-patterns["tenant-1-3"]: Creation complete after 0s [id=index-pattern:tenant-1-3]
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to module.opensearch.opensearch_dashboard_object.tenant-2-index-patterns["tenant-2-2"], provider "provider[\"registry.opentofu.org/opensearch-project/opensearch\"]" produced an unexpected new value: Root
│ object was present, but now absent.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to module.opensearch.opensearch_dashboard_object.tenant-2-index-patterns["tenant-2-3"], provider "provider[\"registry.opentofu.org/opensearch-project/opensearch\"]" produced an unexpected new value: Root
│ object was present, but now absent.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Checking the index-patterns through the opensearch-dashboards UI, I could see, that for tenant-1 the index-pattern tenant-2-3 was added which is not defined in the terraform configuration:

When trying to tofu apply again, it is trying to recreate (already some existing) resources.
The behaviour is kindoff randomly, sometimes it is working, sometimes other index-patterns are assigned to the wrong tenant.

tofu destroy won't delete the wrong index-patterns

What is the expected behavior?

Each index-pattern is assigned to its correct tenant.

What is your host/environment?

Opentofu 1.6.2
opensearch-terraform-provider: 2.3.1
opensearch + opensearch-dashboards: 2.11 (via docker), was reproduceable with opensearch 2.15, too)

Do you have any additional context?

• I think I found a workaround by disabling the parallelism from terraform / opentofu:
  tofu apply -parallelism=1.
  Maybe this workaround gives a hint, that there is an issue with shared resources (in this case tenant) inside the provider.

[Divyaasm]

@prudhvigodithi , @peterzhuamazon Can you please take a look into the issue.

[prudhvigodithi]

[Traige]
Hey @mmuel42, here are some examples created by @rblcoder https://github.com/rblcoder/terraform-opensearch-samples/blob/main/tenant/index_pattern/main.tf for creating tenant, can you please check and see if there is any error with your configuration while I take a look at it.
Thank you
@getsaurabh02