Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add support for not checking callabacks before executing #40

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

tsvetkov-ii
Copy link

Airflow declarative supports check callback for importable. If i have different environments for airflow workers i can't build my dags. I want the solution which can work with checking importable callbacks and not. So i add optional flag for open interface of airflow-declarative.

And i change declarative trafaret schema of constants to dynamically chema with functions for purpose of simple changing schema node with config that we can apply on schema validation.

@KostyaEsmukov
Copy link
Member

If i have different environments for airflow workers i can't build my dags. I want the solution which can work with checking importable callbacks and not.

Each airflow worker (assuming celery executor) has a queue, and a worker might receive tasks for any callable which is expected to run in that queue. But callables from other queues will never be executed by that worker and indeed might be not importable and this is okay.

So would it be possible to automatically limit the callback checking to those tasks set to a specific queue which the current Airflow worker is listening to instead of disabling the check completely?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants