Automating the Extraction of dashboard.json in Terraform

[ebattat]

The problem:

$ make > dashboard.json
Generate the following output:

data.jsonnet_file.dashboard: Reading...
data.jsonnet_file.dashboard: Read complete after 4s [id=fff343c4f694714716c088faa4aff66dbc2f1e763f80e149f87fdd85451cc933]
No changes. Your infrastructure matches the configuration.
Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Outputs:
dashboard = <<EOT
{
   "description": "Dashboard for Faro",
   ....
   "title": "Faro dashboard"
}

EOT

To obtain a clean dashboard JSON, we execute the following code:

with open(sys.argv[1], 'r') as f:
    match = re.search(r'dashboard = <<EOT(.*?)EOT', f.read(), re.DOTALL)
    if match:
        extracted_content = match.group(1).strip()

This method may not be optimal for extracting dashboard.json, as parsing could fail if the output contains 'EOT'.

Do you have a safer method to automate the extraction of dashboard.json in Terraform?

[malcolmholmes]

You need to show us your jsonnet.

But rather than saying:

{
  dashboard: g.dashboard.new(...)
}

Just say:

dashboard.new(...)

[ebattat]

its not working with existing example

[Duologic]

The existing example works quite well for me, can you describe the steps to reproduce with the existing example?

[ebattat]

I run the following steps:

1. Update Makefile with ' -auto-approve'

.PHONY: apply
apply:
	@cd jsonnet/ && \
		jb install && \
		cd ../ && \
		terraform apply -auto-approve

2. terraform init
3. make > dashboard.json
   When opening dashboard.json:

You can apply this plan to save these new output values to the Terraform
state, without changing any real infrastructure.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

dashboard = <<EOT
{
   "description": "Dashboard for Faro",
   "graphTooltip": 1,
   "panels": [
      {
         "datasource": {
            "type": "datasource",
            "uid": "-- Mixed --"
         },
         "fieldConfig": {
            "defaults": {
               "unit": "reqps"
            }
         },
         "gridPos": {
            "h": 8,
            "w": 24
         },
         "id": 1,
         "targets": [
            {
               "datasource": {
                  "type": "prometheus",
                  "uid": "ops-cortex"
               },
               "expr": "sum by (status_code) (rate(request_duration_seconds_count{job=~\".*/faro-api\"}[$__rate_interval]))"
            }
         ],
         "title": "Requests / sec",
         "type": "timeseries"
      }
   ],
   "schemaVersion": 36,
   "time": {
      "from": "now-6h",
      "to": "now"
   },
   "timezone": "utc",
   "title": "Faro dashboard"
}

EOT

[Duologic]

Ah, I think I understand you now, you are using Terraform to render the jsonnet and then attempt to read the rendered JSON from the Terraform output.

Perhaps you can just execute the jsonnet without going through Terraform? Or you could use the local_file resource in Terraform to write files?

[ebattat]

Yes, the following working to me:
jsonnet -J vendor main.libsonnet > /app/dashboard.json

But I think to use Terraform is recommended way because its auto ident the output json.

[Duologic]

I get clear indentation with jsonnet too, not sure what you mean.

[ebattat]

Did you try to run your terraform example ?

[Duologic]

Yes, I get the same results, what's the problem with that?

[Duologic]

I can't really change how Terraform outputs work, that is out of scope for this project.

[malcolmholmes]

Oh, I think i get it. Instead of using terraform's output functionality, you are supposed to use the generated json in the Grafana Terraform provider, in a grafana_dashboard resource to deploy it to Grafana.

As @Duologic suggests, this is working as expected. I'd encourage you to try the Grafana Terraform provider, and experiment more with Terraform.

[ebattat]

We plan to build E2E flow to deploy auto Grafana dashboard automatically using GitHub Actions.
Currently the provided output cannot be used automatically because we need to extract the json from it.

data.jsonnet_file.dashboard: Reading...
data.jsonnet_file.dashboard: Read complete after 4s [id=fff343c4f694714716c088faa4aff66dbc2f1e763f80e149f87fdd85451cc933]
No changes. Your infrastructure matches the configuration.
Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Outputs:
dashboard = <<EOT
{
   "description": "Dashboard for Faro",
   ....
   "title": "Faro dashboard"
}

EOT

I thought that there is a way to get only the json from terraform output instead of using the following code:

with open(sys.argv[1], 'r') as f:
    match = re.search(r'dashboard = <<EOT(.*?)EOT', f.read(), re.DOTALL)
    if match:
        extracted_content = match.group(1).strip()

Is it more clear now ?

[malcolmholmes]

If you just want the json, don't use Terraform. Just invoke jsonnet directly. Using Terraform to invoke the jsonnet you are just making life harder for yourself.