Databricks

[nagydavid]

I am trying to validate a bundle through CI pipeline with this variable

  default_cluster_config:
    description: The default cluster configuration for jobs
    type: complex
    default:
      num_workers: 2
      spark_version: 15.4.x-cpu-ml-scala2.12
      node_type_id: Standard_D3_v2
      data_security_mode: "SINGLE_USER"

and I am getting this error

Error: no value assigned to required variable default_cluster_config. Assignment can be done through the "--var" flag or by setting the BUNDLE_VAR_default_cluster_config environment variable

when I am deploying to the same target, but locally, using the same service principal profile, it works.

  integration_test:
    name: Integration Tests
    needs: file_existence
    if: ${{ needs.file_existence.outputs.run_type == 'unit_and_integration' || needs.file_existence.outputs.run_type == 'integration' }}
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repo
        uses: actions/checkout@v4
      - name: Setup Databricks CLI
        uses: databricks/setup-cli@main
      - name: Validate Bundle For Test Deployment Target in Staging Workspace
        id: validate
        run: |
          databricks bundle validate -t test
      - name: Deploy Bundle to Test Deployment Target in Staging Workspace
        id: deploy
        run: |
          databricks bundle deploy -t test
      - name: Get bundle summary and extract job names
        id: get_summary_and_jobs
        run: |
          databricks bundle summary -t test --output JSON > bundle_summary.json
          job_names="$(jq -r '.resources.jobs | to_entries[] | select(.value.deployment.metadata_file_path != null and .key != "table_properties_job") | .key' bundle_summary.json | paste -sd "," -)"
          echo "job_names=${job_names}" >> "$GITHUB_ENV"
      - name: Check and run deployed jobs
        run: |
          IFS=',' read -ra job_names <<< "${job_names}"
          for job_name in "${job_names[@]}"; do
            databricks bundle run "${job_name}" -t test
          done

      - name: Run table properties job
        run: |
          databricks bundle run "table_properties_job" -t test

the bundle summary

{
  "variables": {
    "default_cluster_config": {
      "type": "complex",
      "default": {
        "data_security_mode": "SINGLE_USER",
        "node_type_id": "Standard_D3_v2",
        "num_workers": 2,
        "spark_version": "15.4.x-cpu-ml-scala2.12"
      },
      "description": "The default cluster configuration for jobs",
      "value": {
        "data_security_mode": "SINGLE_USER",
        "node_type_id": "Standard_D3_v2",
        "num_workers": 2,
        "spark_version": "15.4.x-cpu-ml-scala2.12"
      }
    }, 
    ...