Generating multiple DAGs with Cross DAG Dependencies

[ramnathv]

Thanks for a really awesome demo of how to integrate dbt with airflow. We use airflow extensively and generate multiple DAGs, one for each functional area (marketing, finance, etc.). While we try to keep the DAGs as independent as possible, realistically, there are always a few cross-dag dependencies that we handle using ExternalTaskSensor. Is it possible to handle it within this architecture?

I tried to add tags to the tasks for each DAG. However, except for staging instantiating the DAG with dbt_tag resulted in a DAG with empty dbt_run and dbt_test task groups. Intuitively, here is what I am thinking should work:

1. Use tags to identify separate DAGs.
2. While processing a specific DAG, generate wait_for tasks using ExternalSensor for dependencies that are NOT in the same DAG.

I would love to get your thoughts on the best way to operationalize this.