[ISSUE] Issue with databricks_library resource not starting the cluster before library installation and as a result getting failed

[sudhir-garg]

Configuration

resource "databricks_library" "pyyaml" {
  cluster_id = databricks_cluster.my_cluster_with_libs.id
  pypi {
    package = "PyYAML==6.0.2"
  }
}

Expected Behavior

As per the documentation the resource will first start the cluster if it is not in 'start' state and then install the required packages onto the cluster.

Actual Behavior

The resource is not starting my cluster though it is in 'terminated' state and keep on waiting for the library installation for 15 minutes and then gets failed.

Plan: 1 to add, 0 to change, 0 to destroy.
databricks_library.pyyaml: Creating...
databricks_library.pyyaml: Still creating... [10s elapsed]
databricks_library.pyyaml: Still creating... [20s elapsed]
databricks_library.pyyaml: Still creating... [30s elapsed]
databricks_library.pyyaml: Still creating... [40s elapsed]
databricks_library.pyyaml: Still creating... [50s elapsed]
databricks_library.pyyaml: Still creating... [1m0s elapsed]
databricks_library.pyyaml: Still creating... [1m10s elapsed]
databricks_library.pyyaml: Still creating... [1m20s elapsed]
databricks_library.pyyaml: Still creating... [1m30s elapsed]
databricks_library.pyyaml: Still creating... [1m40s elapsed]
databricks_library.pyyaml: Still creating... [1m50s elapsed]
databricks_library.pyyaml: Still creating... [2m0s elapsed]
databricks_library.pyyaml: Still creating... [2m10s elapsed]
databricks_library.pyyaml: Still creating... [2m20s elapsed]
databricks_library.pyyaml: Still creating... [2m30s elapsed]
databricks_library.pyyaml: Still creating... [2m40s elapsed]
databricks_library.pyyaml: Still creating... [2m50s elapsed]
databricks_library.pyyaml: Still creating... [3m0s elapsed]
databricks_library.pyyaml: Still creating... [3m10s elapsed]
databricks_library.pyyaml: Still creating... [3m20s elapsed]
databricks_library.pyyaml: Still creating... [3m30s elapsed]
databricks_library.pyyaml: Still creating... [3m40s elapsed]
databricks_library.pyyaml: Still creating... [3m50s elapsed]
databricks_library.pyyaml: Still creating... [4m0s elapsed]
databricks_library.pyyaml: Still creating... [4m10s elapsed]
databricks_library.pyyaml: Still creating... [4m20s elapsed]
databricks_library.pyyaml: Still creating... [4m30s elapsed]
databricks_library.pyyaml: Still creating... [4m40s elapsed]
databricks_library.pyyaml: Still creating... [4m50s elapsed]
databricks_library.pyyaml: Still creating... [5m0s elapsed]
databricks_library.pyyaml: Still creating... [5m10s elapsed]
databricks_library.pyyaml: Still creating... [5m20s elapsed]
databricks_library.pyyaml: Still creating... [5m30s elapsed]
databricks_library.pyyaml: Still creating... [5m40s elapsed]
databricks_library.pyyaml: Still creating... [5m50s elapsed]
databricks_library.pyyaml: Still creating... [6m0s elapsed]
databricks_library.pyyaml: Still creating... [6m10s elapsed]
databricks_library.pyyaml: Still creating... [6m20s elapsed]
databricks_library.pyyaml: Still creating... [6m30s elapsed]
databricks_library.pyyaml: Still creating... [6m40s elapsed]
databricks_library.pyyaml: Still creating... [6m50s elapsed]
databricks_library.pyyaml: Still creating... [7m0s elapsed]
databricks_library.pyyaml: Still creating... [7m10s elapsed]
databricks_library.pyyaml: Still creating... [7m20s elapsed]
databricks_library.pyyaml: Still creating... [7m30s elapsed]
databricks_library.pyyaml: Still creating... [7m40s elapsed]
databricks_library.pyyaml: Still creating... [7m50s elapsed]
databricks_library.pyyaml: Still creating... [8m0s elapsed]
databricks_library.pyyaml: Still creating... [8m10s elapsed]
databricks_library.pyyaml: Still creating... [8m20s elapsed]
databricks_library.pyyaml: Still creating... [8m30s elapsed]
databricks_library.pyyaml: Still creating... [8m40s elapsed]
databricks_library.pyyaml: Still creating... [8m50s elapsed]
databricks_library.pyyaml: Still creating... [9m0s elapsed]
databricks_library.pyyaml: Still creating... [9m10s elapsed]
databricks_library.pyyaml: Still creating... [9m20s elapsed]
databricks_library.pyyaml: Still creating... [9m30s elapsed]
databricks_library.pyyaml: Still creating... [9m40s elapsed]
databricks_library.pyyaml: Still creating... [9m50s elapsed]
databricks_library.pyyaml: Still creating... [10m0s elapsed]
databricks_library.pyyaml: Still creating... [10m10s elapsed]
databricks_library.pyyaml: Still creating... [10m20s elapsed]
databricks_library.pyyaml: Still creating... [10m30s elapsed]
databricks_library.pyyaml: Still creating... [10m40s elapsed]
databricks_library.pyyaml: Still creating... [10m50s elapsed]
databricks_library.pyyaml: Still creating... [11m0s elapsed]
databricks_library.pyyaml: Still creating... [11m10s elapsed]
databricks_library.pyyaml: Still creating... [11m20s elapsed]
databricks_library.pyyaml: Still creating... [11m30s elapsed]
databricks_library.pyyaml: Still creating... [11m40s elapsed]
databricks_library.pyyaml: Still creating... [11m50s elapsed]
databricks_library.pyyaml: Still creating... [12m0s elapsed]
databricks_library.pyyaml: Still creating... [12m10s elapsed]
databricks_library.pyyaml: Still creating... [12m20s elapsed]
databricks_library.pyyaml: Still creating... [12m30s elapsed]
databricks_library.pyyaml: Still creating... [12m40s elapsed]
databricks_library.pyyaml: Still creating... [12m50s elapsed]
databricks_library.pyyaml: Still creating... [13m0s elapsed]
databricks_library.pyyaml: Still creating... [13m10s elapsed]
databricks_library.pyyaml: Still creating... [13m20s elapsed]
databricks_library.pyyaml: Still creating... [13m30s elapsed]
databricks_library.pyyaml: Still creating... [13m40s elapsed]
databricks_library.pyyaml: Still creating... [13m50s elapsed]
databricks_library.pyyaml: Still creating... [14m0s elapsed]
databricks_library.pyyaml: Still creating... [14m10s elapsed]
databricks_library.pyyaml: Still creating... [14m20s elapsed]
databricks_library.pyyaml: Still creating... [14m30s elapsed]
databricks_library.pyyaml: Still creating... [14m40s elapsed]
databricks_library.pyyaml: Still creating... [14m50s elapsed]
databricks_library.pyyaml: Still creating... [15m0s elapsed]
╷
│ Error: failed to wait for library installation
│ 
│   with databricks_library.pyyaml,
│   on dbx_clusters.tf line 68, in resource "databricks_library" "pyyaml":
│   68: resource "databricks_library" "pyyaml" {
│ 
│ 0 libraries are ready, but there are still 1 pending
╵

Exited with code exit status 1

Steps to Reproduce

Terraform and provider versions

Terraform version : Terraform v1.8.0
databricks provider version: databricks/databricks v1.62.0

Is it a regression?

Debug Output

Important Factoids

the databricks version 1.56.0 seems to be working fine (didn't check the versions in between). When I used this version then as per expectations it first start the cluster and then deployed the required packages.

Would you like to implement a fix?