Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add example to demonstrate a job with condition task #20

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions knowledge_base/job_with_condition_task/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
.databricks
31 changes: 31 additions & 0 deletions knowledge_base/job_with_condition_task/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# job_with_condition_task

Example to show how a "condition task" can be used.

## Usage

Configure the workspace to use:
```sh
export DATABRICKS_CONFIG_PROFILE="<workspace profile>"
```

Modify the SQL warehouse name in the configuration to a running SQL warehouse in your workspace.
You can list the SQL warehouses in your workspace using the following command:
```sh
databricks warehouses list
```

Deploy the bundle:
```sh
databricks bundle deploy
```

Run and observe the SQL query task is **not** executed:
```sh
databricks bundle run example_job
```

Run and observe the SQL query task is executed:
```sh
databricks bundle run example_job --params "run_sql_task=true"
```
38 changes: 38 additions & 0 deletions knowledge_base/job_with_condition_task/databricks.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
bundle:
name: job_with_condition_task

variables:
warehouse_id:
lookup:
warehouse: "Shared Warehouse"

resources:
jobs:
example_job:
name: "Example job to demonstrate the use of a condition task"

tasks:
- task_key: condition
condition_task:
left: "{{ job.parameters.[run_sql_task] }}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we show an example with a task value instead? So we have one Python notebook compute something, return it, and then use it in the conditional?

(Btw we want to support that use case for SQL queries too but don't have that support yet.)

The current example using a job parameter is a bit atypical.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could that be an additional example? For this one, I wanted to repro an issue and capture the example.

Copy link

@gaborratky-db gaborratky-db Feb 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could have a notebook "process_data" that does some data processing and sets a task value "optimize" that determines whether to run OPTIMIZE <table> in a subsequent SQL query task:

if {{ tasks.process_data.values.optimize }} == "true" => OPTIMIZE <table>

and do nothing if false

right: "true"
op: EQUAL_TO

- task_key: run_query
depends_on:
- task_key: condition
outcome: "true"

sql_task:
warehouse_id: ${var.warehouse_id}
file:
path: ./query.sql

parameters:
- name: "run_sql_task"
default: "false"

targets:
dev:
default: true
mode: development
2 changes: 2 additions & 0 deletions knowledge_base/job_with_condition_task/query.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
-- Dummy query to test the job with condition task
select * from range(10)