forked from apache/airflow
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathCHANGELOG.txt
4724 lines (4555 loc) · 268 KB
/
CHANGELOG.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Airflow 2.1.1, 2021-07-02
-------------------------
Bug Fixes
"""""""""
- Don't crash attempting to mask secrets in dict with non-string keys (#16601)
- Always install sphinx_airflow_theme from ``PyPI`` (#16594)
- Remove limitation for elasticsearch library (#16553)
- Adding extra requirements for build and runtime of the PROD image. (#16170)
- Cattrs 1.7.0 released by the end of May 2021 break lineage usage (#16173)
- Removes unnecessary packages from setup_requires (#16139)
- Pins docutils to <0.17 until breaking behaviour is fixed (#16133)
- Improvements for Docker Image docs (#14843)
- Ensure that ``dag_run.conf`` is a dict (#15057)
- Fix CLI connections import and migrate logic from secrets to Connection model (#15425)
- Fix Dag Details start date bug (#16206)
- Fix DAG run state not updated while DAG is paused (#16343)
- Allow null value for operator field in task_instance schema(REST API) (#16516)
- Avoid recursion going too deep when redacting logs (#16491)
- Backfill: Don't create a DagRun if no tasks match task regex (#16461)
- Tree View UI for larger DAGs & more consistent spacing in Tree View (#16522)
- Correctly handle None returns from Query.scalar() (#16345)
- Adding ``only_active`` parameter to /dags endpoint (#14306)
- Don't show stale Serialized DAGs if they are deleted in DB (#16368)
- Make REST API List DAGs endpoint consistent with UI/CLI behaviour (#16318)
- Support remote logging in elasticsearch with ``filebeat 7`` (#14625)
- Queue tasks with higher priority and earlier execution_date first. (#15210)
- Make task ID on legend have enough width and width of line chart to be 100%. (#15915)
- Fix normalize-url vulnerability (#16375)
- Validate retries value on init for better errors (#16415)
- add num_runs query param for tree refresh (#16437)
- Fix templated default/example values in config ref docs (#16442)
- Add ``passphrase`` and ``private_key`` to default sensitive field names (#16392)
- Fix tasks in an infinite slots pool were never scheduled (#15247)
- Fix Orphaned tasks stuck in CeleryExecutor as running (#16550)
- Don't fail to log if we can't redact something (#16118)
- Set max tree width to 1200 pixels (#16067)
- Fill the "job_id" field for ``airflow task run`` without ``--local``/``--raw`` for KubeExecutor (#16108)
- Fixes problem where conf variable was used before initialization (#16088)
- Fix apply defaults for task decorator (#16085)
- Parse recently modified files even if just parsed (#16075)
- Ensure that we don't try to mask empty string in logs (#16057)
- Don't die when masking ``log.exception`` when there is no exception (#16047)
- Restores apply_defaults import in base_sensor_operator (#16040)
- Fix auto-refresh in tree view When webserver ui is not in ``/`` (#16018)
- Fix dag.clear() to set multiple dags to running when necessary (#15382)
- Fix Celery executor getting stuck randomly because of reset_signals in multiprocessing (#15989)
Airflow 2.1.0, 2021-05-21
-------------------------
New Features
""""""""""""
- Add ``PythonVirtualenvDecorator`` to Taskflow API (#14761)
- Add ``Taskgroup`` decorator (#15034)
- Create a DAG Calendar View (#15423)
- Create cross-DAG dependencies view (#13199)
- Add rest API to query for providers (#13394)
- Mask passwords and sensitive info in task logs and UI (#15599)
- Add ``SubprocessHook`` for running commands from operators (#13423)
- Add DAG Timeout in UI page "DAG Details" (#14165)
- Add ``WeekDayBranchOperator`` (#13997)
- Add JSON linter to DAG Trigger UI (#13551)
- Add DAG Description Doc to Trigger UI Page (#13365)
- Add airflow webserver URL into SLA miss email. (#13249)
- Add read only REST API endpoints for users (#14735)
- Add files to generate Airflow's Python SDK (#14739)
- Add dynamic fields to snowflake connection (#14724)
- Add read only REST API endpoint for roles and permissions (#14664)
- Add new datetime branch operator (#11964)
- Add Google leveldb hook and operator (#13109) (#14105)
- Add plugins endpoint to the REST API (#14280)
- Add ``worker_pod_pending_timeout`` support (#15263)
- Add support for labeling DAG edges (#15142)
- Add CUD REST API endpoints for Roles (#14840)
- Import connections from a file (#15177)
- A bunch of ``template_fields_renderers`` additions (#15130)
- Add REST API query sort and order to some endpoints (#14895)
- Add timezone context in new ui (#15096)
- Add query mutations to new UI (#15068)
- Add different modes to sort dag files for parsing (#15046)
- Auto refresh on Tree View (#15474)
- BashOperator to raise ``AirflowSkipException`` on exit code 99 (by default, configurable) (#13421) (#14963)
- Clear tasks by task ids in REST API (#14500)
- Support jinja2 native Python types (#14603)
- Allow celery workers without gossip or mingle modes (#13880)
- Add ``airflow jobs check`` CLI command to check health of jobs (Scheduler etc) (#14519)
- Rename ``DateTimeBranchOperator`` to ``BranchDateTimeOperator`` (#14720)
Improvements
""""""""""""
- Add optional result handler callback to ``DbApiHook`` (#15581)
- Update Flask App Builder limit to recently released 3.3 (#15792)
- Prevent creating flask sessions on REST API requests (#15295)
- Sync DAG specific permissions when parsing (#15311)
- Increase maximum length of pool name on Tasks to 256 characters (#15203)
- Enforce READ COMMITTED isolation when using mysql (#15714)
- Auto-apply ``apply_default`` to subclasses of ``BaseOperator`` (#15667)
- Emit error on duplicated DAG ID (#15302)
- Update ``KubernetesExecutor`` pod templates to allow access to IAM permissions (#15669)
- More verbose logs when running ``airflow db check-migrations`` (#15662)
- When one_success mark task as failed if no success (#15467)
- Add an option to trigger a dag w/o changing conf (#15591)
- Add Airflow UI instance_name configuration option (#10162)
- Add a decorator to retry functions with DB transactions (#14109)
- Add return to PythonVirtualenvOperator's execute method (#14061)
- Add verify_ssl config for kubernetes (#13516)
- Add description about ``secret_key`` when Webserver > 1 (#15546)
- Add Traceback in LogRecord in ``JSONFormatter`` (#15414)
- Add support for arbitrary json in conn uri format (#15100)
- Adds description field in variable (#12413) (#15194)
- Add logs to show last modified in SFTP, FTP and Filesystem sensor (#15134)
- Execute ``on_failure_callback`` when SIGTERM is received (#15172)
- Allow hiding of all edges when highlighting states (#15281)
- Display explicit error in case UID has no actual username (#15212)
- Serve logs with Scheduler when using Local or Sequential Executor (#15557)
- Deactivate trigger, refresh, and delete controls on dag detail view. (#14144)
- Turn off autocomplete for connection forms (#15073)
- Increase default ``worker_refresh_interval`` to ``6000`` seconds (#14970)
- Only show User's local timezone if it's not UTC (#13904)
- Suppress LOG/WARNING for a few tasks CLI for better CLI experience (#14567)
- Configurable API response (CORS) headers (#13620)
- Allow viewers to see all docs links (#14197)
- Update Tree View date ticks (#14141)
- Make the tooltip to Pause / Unpause a DAG clearer (#13642)
- Warn about precedence of env var when getting variables (#13501)
- Move ``[celery] default_queue`` config to ``[operators] default_queue`` to re-use between executors (#14699)
Bug Fixes
"""""""""
- Fix 500 error from ``updateTaskInstancesState`` API endpoint when ``dry_run`` not passed (#15889)
- Ensure that task preceding a PythonVirtualenvOperator doesn't fail (#15822)
- Prevent mixed case env vars from crashing processes like worker (#14380)
- Fixed type annotations in DAG decorator (#15778)
- Fix on_failure_callback when task receive SIGKILL (#15537)
- Fix dags table overflow (#15660)
- Fix changing the parent dag state on subdag clear (#15562)
- Fix reading from zip package to default to text (#13962)
- Fix wrong parameter for ``drawDagStatsForDag`` in dags.html (#13884)
- Fix QueuedLocalWorker crashing with EOFError (#13215)
- Fix typo in ``NotPreviouslySkippedDep`` (#13933)
- Fix parallelism after KubeExecutor pod adoption (#15555)
- Fix kube client on mac with keepalive enabled (#15551)
- Fixes wrong limit for dask for python>3.7 (should be <3.7) (#15545)
- Fix Task Adoption in ``KubernetesExecutor`` (#14795)
- Fix timeout when using XCom with ``KubernetesPodOperator`` (#15388)
- Fix deprecated provider aliases in "extras" not working (#15465)
- Fixed default XCom deserialization. (#14827)
- Fix used_group_ids in ``dag.partial_subset`` (#13700) (#15308)
- Further fix trimmed ``pod_id`` for ``KubernetesPodOperator`` (#15445)
- Bugfix: Invalid name when trimmed `pod_id` ends with hyphen in ``KubernetesPodOperator`` (#15443)
- Fix incorrect slots stats when TI ``pool_slots > 1`` (#15426)
- Fix DAG last run link (#15327)
- Fix ``sync-perm`` to work correctly when update_fab_perms = False (#14847)
- Fixes limits on Arrow for plexus test (#14781)
- Fix UI bugs in tree view (#14566)
- Fix AzureDataFactoryHook failing to instantiate its connection (#14565)
- Fix permission error on non-POSIX filesystem (#13121)
- Fix spelling in "ignorable" (#14348)
- Fix get_context_data doctest import (#14288)
- Correct typo in ``GCSObjectsWtihPrefixExistenceSensor`` (#14179)
- Fix order of failed deps (#14036)
- Fix critical ``CeleryKubernetesExecutor`` bug (#13247)
- Fix four bugs in ``StackdriverTaskHandler`` (#13784)
- ``func.sum`` may return ``Decimal`` that break rest APIs (#15585)
- Persist tags params in pagination (#15411)
- API: Raise ``AlreadyExists`` exception when the ``execution_date`` is same (#15174)
- Remove duplicate call to ``sync_metadata`` inside ``DagFileProcessorManager`` (#15121)
- Extra ``docker-py`` update to resolve docker op issues (#15731)
- Ensure executors end method is called (#14085)
- Remove ``user_id`` from API schema (#15117)
- Prevent clickable bad links on disabled pagination (#15074)
- Acquire lock on db for the time of migration (#10151)
- Skip SLA check only if SLA is None (#14064)
- Print right version in airflow info command (#14560)
- Make ``airflow info`` work with pipes (#14528)
- Rework client-side script for connection form. (#14052)
- API: Add ``CollectionInfo`` in all Collections that have ``total_entries`` (#14366)
- Fix ``task_instance_mutation_hook`` when importing airflow.models.dagrun (#15851)
Doc only changes
""""""""""""""""
- Fix docstring of SqlSensor (#15466)
- Small changes on "DAGs and Tasks documentation" (#14853)
- Add note on changes to configuration options (#15696)
- Add docs to the ``markdownlint`` and ``yamllint`` config files (#15682)
- Rename old "Experimental" API to deprecated in the docs. (#15653)
- Fix documentation error in `git_sync_template.yaml` (#13197)
- Fix doc link permission name (#14972)
- Fix link to Helm chart docs (#14652)
- Fix docstrings for Kubernetes code (#14605)
- docs: Capitalize & minor fixes (#14283) (#14534)
- Fixed reading from zip package to default to text. (#13984)
- An initial rework of the "Concepts" docs (#15444)
- Improve docstrings for various modules (#15047)
- Add documentation on database connection URI (#14124)
- Add Helm Chart logo to docs index (#14762)
- Create a new documentation package for Helm Chart (#14643)
- Add docs about supported logging levels (#14507)
- Update docs about tableau and salesforce provider (#14495)
- Replace deprecated doc links to the correct one (#14429)
- Refactor redundant doc url logic to use utility (#14080)
- docs: NOTICE: Updated 2016-2019 to 2016-now (#14248)
- Skip DAG perm sync during parsing if possible (#15464)
- Add picture and examples for Edge Labels (#15310)
- Add example DAG & how-to guide for sqlite (#13196)
- Add links to new modules for deprecated modules (#15316)
- Add note in Updating.md about FAB data model change (#14478)
Misc/Internal
"""""""""""""
- Fix ``logging.exception`` redundancy (#14823)
- Bump ``stylelint`` to remove vulnerable sub-dependency (#15784)
- Add resolution to force dependencies to use patched version of lodash (#15777)
- Update croniter to 1.0.x series (#15769)
- Get rid of Airflow 1.10 in Breeze (#15712)
- Run helm chart tests in parallel (#15706)
- Bump ``ssri`` from 6.0.1 to 6.0.2 in /airflow/www (#15437)
- Remove the limit on Gunicorn dependency (#15611)
- Better "dependency already registered" warning message for tasks #14613 (#14860)
- Pin pandas-gbq to <0.15.0 (#15114)
- Use Pip 21.* to install airflow officially (#15513)
- Bump mysqlclient to support the 1.4.x and 2.x series (#14978)
- Finish refactor of DAG resource name helper (#15511)
- Refactor/Cleanup Presentation of Graph Task and Path Highlighting (#15257)
- Standardize default fab perms (#14946)
- Remove ``datepicker`` for task instance detail view (#15284)
- Turn provider's import warnings into debug logs (#14903)
- Remove left-over fields from required in provider_info schema. (#14119)
- Deprecate ``tableau`` extra (#13595)
- Use built-in `cached_property` on Python 3.8 where possible (#14606)
- Clean-up JS code in UI templates (#14019)
- Bump elliptic from 6.5.3 to 6.5.4 in /airflow/www (#14668)
- Switch to f-strings using ``flynt``. (#13732)
- use ``jquery`` ready instead of vanilla js (#15258)
- Migrate task instance log (ti_log) js (#15309)
- Migrate graph js (#15307)
- Migrate dags.html javascript (#14692)
- Removes unnecessary AzureContainerInstance connection type (#15514)
- Separate Kubernetes pod_launcher from core airflow (#15165)
- update remaining old import paths of operators (#15127)
- Remove broken and undocumented "demo mode" feature (#14601)
- Simplify configuration/legibility of ``Webpack`` entries (#14551)
- remove inline tree js (#14552)
- Js linting and inline migration for simple scripts (#14215)
- Remove use of repeated constant in AirflowConfigParser (#14023)
- Deprecate email credentials from environment variables. (#13601)
- Remove unused 'context' variable in task_instance.py (#14049)
- Disable suppress_logs_and_warning in cli when debugging (#13180)
Airflow 2.0.2, 2021-04-19
-------------------------
Bug Fixes
"""""""""
* Bugfix: ``TypeError`` when Serializing & sorting iterable properties of DAGs (#15395)
* Fix missing ``on_load`` trigger for folder-based plugins (#15208)
* ``kubernetes cleanup-pods`` subcommand will only clean up Airflow-created Pods (#15204)
* Fix password masking in CLI action_logging (#15143)
* Fix url generation for TriggerDagRunOperatorLink (#14990)
* Restore base lineage backend (#14146)
* Unable to trigger backfill or manual jobs with Kubernetes executor. (#14160)
* Bugfix: Task docs are not shown in the Task Instance Detail View (#15191)
* Bugfix: Fix overriding ``pod_template_file`` in KubernetesExecutor (#15197)
* Bugfix: resources in ``executor_config`` breaks Graph View in UI (#15199)
* Fix celery executor bug trying to call len on map (#14883)
* Fix bug in airflow.stats timing that broke dogstatsd mode (#15132)
* Avoid scheduler/parser manager deadlock by using non-blocking IO (#15112)
* Re-introduce ``dagrun.schedule_delay`` metric (#15105)
* Compare string values, not if strings are the same object in Kube executor(#14942)
* Pass queue to BaseExecutor.execute_async like in airflow 1.10 (#14861)
* Scheduler: Remove TIs from starved pools from the critical path. (#14476)
* Remove extra/needless deprecation warnings from airflow.contrib module (#15065)
* Fix support for long dag_id and task_id in KubernetesExecutor (#14703)
* Sort lists, sets and tuples in Serialized DAGs (#14909)
* Simplify cleaning string passed to origin param (#14738) (#14905)
* Fix error when running tasks with Sentry integration enabled. (#13929)
* Webserver: Sanitize string passed to origin param (#14738)
* Fix losing duration < 1 secs in tree (#13537)
* Pin SQLAlchemy to <1.4 due to breakage of sqlalchemy-utils (#14812)
* Fix KubernetesExecutor issue with deleted pending pods (#14810)
* Default to Celery Task model when backend model does not exist (#14612)
* Bugfix: Plugins endpoint was unauthenticated (#14570)
* BugFix: fix DAG doc display (especially for TaskFlow DAGs) (#14564)
* BugFix: TypeError in airflow.kubernetes.pod_launcher's monitor_pod (#14513)
* Bugfix: Fix wrong output of tags and owners in dag detail API endpoint (#14490)
* Fix logging error with task error when JSON logging is enabled (#14456)
* Fix statsd metrics not sending when using daemon mode (#14454)
* Gracefully handle missing start_date and end_date for DagRun (#14452)
* BugFix: Serialize max_retry_delay as a timedelta (#14436)
* Fix crash when user clicks on "Task Instance Details" caused by start_date being None (#14416)
* BugFix: Fix TaskInstance API call fails if a task is removed from running DAG (#14381)
* Scheduler should not fail when invalid ``executor_config`` is passed (#14323)
* Fix bug allowing task instances to survive when dagrun_timeout is exceeded (#14321)
* Fix bug where DAG timezone was not always shown correctly in UI tooltips (#14204)
* Use ``Lax`` for ``cookie_samesite`` when empty string is passed (#14183)
* [AIRFLOW-6076] fix ``dag.cli()`` KeyError (#13647)
* Fix running child tasks in a subdag after clearing a successful subdag (#14776)
Improvements
""""""""""""
* Remove unused JS packages causing false security alerts (#15383)
* Change default of ``[kubernetes] enable_tcp_keepalive`` for new installs to ``True`` (#15338)
* Fixed #14270: Add error message in OOM situations (#15207)
* Better compatibility/diagnostics for arbitrary UID in docker image (#15162)
* Updates 3.6 limits for latest versions of a few libraries (#15209)
* Adds Blinker dependency which is missing after recent changes (#15182)
* Remove 'conf' from search_columns in DagRun View (#15099)
* More proper default value for namespace in K8S cleanup-pods CLI (#15060)
* Faster default role syncing during webserver start (#15017)
* Speed up webserver start when there are many DAGs (#14993)
* Much easier to use and better documented Docker image (#14911)
* Use ``libyaml`` C library when available. (#14577)
* Don't create unittest.cfg when not running in unit test mode (#14420)
* Webserver: Allow Filtering TaskInstances by queued_dttm (#14708)
* Update Flask-AppBuilder dependency to allow 3.2 (and all 3.x series) (#14665)
* Remember expanded task groups in browser local storage (#14661)
* Add plain format output to cli tables (#14546)
* Make ``airflow dags show`` command display TaskGroups (#14269)
* Increase maximum size of ``extra`` connection field. (#12944)
* Speed up clear_task_instances by doing a single sql delete for TaskReschedule (#14048)
* Add more flexibility with FAB menu links (#13903)
* Add better description and guidance in case of sqlite version mismatch (#14209)
Doc only changes
""""""""""""""""
* Add documentation create/update community providers (#15061)
* Fix mistake and typos in airflow.utils.timezone docstrings (#15180)
* Replace new url for Stable Airflow Docs (#15169)
* Docs: Clarify behavior of delete_worker_pods_on_failure (#14958)
* Create a documentation package for Docker image (#14846)
* Multiple minor doc (OpenAPI) fixes (#14917)
* Replace Graph View Screenshot to show Auto-refresh (#14571)
Misc/Internal
"""""""""""""
* Import Connection lazily in hooks to avoid cycles (#15361)
* Rename last_scheduler_run into last_parsed_time, and ensure it's updated in DB (#14581)
* Make TaskInstance.pool_slots not nullable with a default of 1 (#14406)
* Log migrations info in consistent way (#14158)
Airflow 2.0.1, 2021-02-08
-------------------------
Bug Fixes
"""""""""
- Bugfix: Return XCom Value in the XCom Endpoint API (#13684)
- Bugfix: Import error when using custom backend and ``sql_alchemy_conn_secret`` (#13260)
- Allow PID file path to be relative when daemonize a process (scheduler, kerberos, etc) (#13232)
- Bugfix: no generic ``DROP CONSTRAINT`` in MySQL during ``airflow db upgrade`` (#13239)
- Bugfix: Sync Access Control defined in DAGs when running ``sync-perm`` (#13377)
- Stop sending Callback Requests if no callbacks are defined on DAG (#13163)
- BugFix: Dag-level Callback Requests were not run (#13651)
- Stop creating duplicate Dag File Processors (#13662)
- Filter DagRuns with Task Instances in removed State while Scheduling (#13165)
- Bump ``datatables.net`` from 1.10.21 to 1.10.22 in /airflow/www (#13143)
- Bump ``datatables.net`` JS to 1.10.23 (#13253)
- Bump ``dompurify`` from 2.0.12 to 2.2.6 in /airflow/www (#13164)
- Update minimum ``cattrs`` version (#13223)
- Remove inapplicable arg 'output' for CLI pools import/export (#13071)
- Webserver: Fix the behavior to deactivate the authentication option and add docs (#13191)
- Fix: add support for no-menu plugin views (#11742)
- Add ``python-daemon`` limit for Python 3.8+ to fix daemon crash (#13540)
- Change the default celery ``worker_concurrency`` to 16 (#13612)
- Audit Log records View should not contain link if ``dag_id`` is None (#13619)
- Fix invalid ``continue_token`` for cleanup list pods (#13563)
- Switches to latest version of snowflake connector (#13654)
- Fix backfill crash on task retry or reschedule (#13712)
- Setting ``max_tis_per_query`` to ``0`` now correctly removes the limit (#13512)
- Fix race conditions in task callback invocations (#10917)
- Fix webserver exiting when gunicorn master crashes (#13518)(#13780)
- Fix SQL syntax to check duplicate connections (#13783)
- ``BaseBranchOperator`` will push to xcom by default (#13704) (#13763)
- Fix Deprecation for ``configuration.getsection`` (#13804)
- Fix TaskNotFound in log endpoint (#13872)
- Fix race condition when using Dynamic DAGs (#13893)
- Fix: Linux/Chrome window bouncing in Webserver
- Fix db shell for sqlite (#13907)
- Only compare updated time when Serialized DAG exists (#13899)
- Fix dag run type enum query for mysqldb driver (#13278)
- Add authentication to lineage endpoint for experimental API (#13870)
- Do not add User role perms to custom roles. (#13856)
- Do not add ``Website.can_read`` access to default roles. (#13923)
- Fix invalid value error caused by long Kubernetes pod name (#13299)
- Fix DB Migration for SQLite to upgrade to 2.0 (#13921)
- Bugfix: Manual DagRun trigger should not skip scheduled runs (#13963)
- Stop loading Extra Operator links in Scheduler (#13932)
- Added missing return parameter in read function of ``FileTaskHandler`` (#14001)
- Bugfix: Do not try to create a duplicate Dag Run in Scheduler (#13920)
- Make ``v1/config`` endpoint respect webserver ``expose_config`` setting (#14020)
- Disable row level locking for Mariadb and MySQL <8 (#14031)
- Bugfix: Fix permissions to triggering only specific DAGs (#13922)
- Fix broken SLA Mechanism (#14056)
- Bugfix: Scheduler fails if task is removed at runtime (#14057)
- Remove permissions to read Configurations for User and Viewer roles (#14067)
- Fix DB Migration from 2.0.1rc1
Improvements
""""""""""""
- Increase the default ``min_file_process_interval`` to decrease CPU Usage (#13664)
- Dispose connections when running tasks with ``os.fork`` & ``CeleryExecutor`` (#13265)
- Make function purpose clearer in ``example_kubernetes_executor`` example dag (#13216)
- Remove unused libraries - ``flask-swagger``, ``funcsigs`` (#13178)
- Display alternative tooltip when a Task has yet to run (no TI) (#13162)
- User werkzeug's own type conversion for request args (#13184)
- UI: Add ``queued_by_job_id`` & ``external_executor_id`` Columns to TI View (#13266)
- Make ``json-merge-patch`` an optional library and unpin it (#13175)
- Adds missing LDAP "extra" dependencies to ldap provider. (#13308)
- Refactor ``setup.py`` to better reflect changes in providers (#13314)
- Pin ``pyjwt`` and Add integration tests for Apache Pinot (#13195)
- Removes provider-imposed requirements from ``setup.cfg`` (#13409)
- Replace deprecated decorator (#13443)
- Streamline & simplify ``__eq__`` methods in models Dag and BaseOperator (#13449)
- Additional properties should be allowed in provider schema (#13440)
- Remove unused dependency - ``contextdecorator`` (#13455)
- Remove 'typing' dependency (#13472)
- Log migrations info in consistent way (#13458)
- Unpin ``mysql-connector-python`` to allow ``8.0.22`` (#13370)
- Remove thrift as a core dependency (#13471)
- Add ``NotFound`` response for DELETE methods in OpenAPI YAML (#13550)
- Stop Log Spamming when ``[core] lazy_load_plugins`` is ``False`` (#13578)
- Display message and docs link when no plugins are loaded (#13599)
- Unpin restriction for ``colorlog`` dependency (#13176)
- Add missing Dag Tag for Example DAGs (#13665)
- Support tables in DAG docs (#13533)
- Add ``python3-openid`` dependency (#13714)
- Add ``__repr__`` for Executors (#13753)
- Add description to hint if ``conn_type`` is missing (#13778)
- Upgrade Azure blob to v12 (#12188)
- Add extra field to ``get_connnection`` REST endpoint (#13885)
- Make Smart Sensors DB Migration idempotent (#13892)
- Improve the error when DAG does not exist when running dag pause command (#13900)
- Update ``airflow_local_settings.py`` to fix an error message (#13927)
- Only allow passing JSON Serializable conf to ``TriggerDagRunOperator`` (#13964)
- Bugfix: Allow getting details of a DAG with null ``start_date`` (REST API) (#13959)
- Add params to the DAG details endpoint (#13790)
- Make the role assigned to anonymous users customizable (#14042)
- Retry critical methods in Scheduler loop in case of ``OperationalError`` (#14032)
Doc only changes
""""""""""""""""
- Add Missing Statsd Metrics in Docs (#13708)
- Add Missing Email configs in Configuration doc (#13709)
- Add quick start for Airflow on Docker (#13660)
- Describe which Python versions are supported (#13259)
- Add note block to 2.x migration docs (#13094)
- Add documentation about webserver_config.py (#13155)
- Add missing version information to recently added configs (#13161)
- API: Use generic information in UpdateMask component (#13146)
- Add Airflow 2.0.0 to requirements table (#13140)
- Avoid confusion in doc for CeleryKubernetesExecutor (#13116)
- Update docs link in REST API spec (#13107)
- Add link to PyPI Repository to provider docs (#13064)
- Fix link to Airflow master branch documentation (#13179)
- Minor enhancements to Sensors docs (#13381)
- Use 2.0.0 in Airflow docs & Breeze (#13379)
- Improves documentation regarding providers and custom connections (#13375)(#13410)
- Fix malformed table in production-deployment.rst (#13395)
- Update celery.rst to fix broken links (#13400)
- Remove reference to scheduler run_duration param in docs (#13346)
- Set minimum SQLite version supported (#13412)
- Fix installation doc (#13462)
- Add docs about mocking variables and connections (#13502)
- Add docs about Flask CLI (#13500)
- Fix Upgrading to 2 guide to use ``rbac`` UI (#13569)
- Make docs clear that Auth can not be disabled for Stable API (#13568)
- Remove archived links from docs & add link for AIPs (#13580)
- Minor fixes in upgrading-to-2.rst (#13583)
- Fix Link in Upgrading to 2.0 guide (#13584)
- Fix heading for Mocking section in best-practices.rst (#13658)
- Add docs on how to use custom operators within plugins folder (#13186)
- Update docs to register Operator Extra Links (#13683)
- Improvements for database setup docs (#13696)
- Replace module path to Class with just Class Name (#13719)
- Update DAG Serialization docs (#13722)
- Fix link to Apache Airflow docs in webserver (#13250)
- Clarifies differences between extras and provider packages (#13810)
- Add information about all access methods to the environment (#13940)
- Docs: Fix FAQ on scheduler latency (#13969)
- Updated taskflow api doc to show dependency with sensor (#13968)
- Add deprecated config options to docs (#13883)
- Added a FAQ section to the Upgrading to 2 doc (#13979)
Airflow 2.0.0, 2020-12-18
----------------------------
The full changelog is about 3,000 lines long (already excluding everything backported to 1.10)
so please check `Airflow 2.0.0 Highligths Blog Post <https://airflow.apache.org/blog/airflow-two-point-oh-is-here/>`_
instead.
Airflow 1.10.15, 2021-03-17
----------------------------
Bug Fixes
"""""""""
- Fix ``airflow db upgrade`` to upgrade db as intended (#13267)
- Moved boto3 limitation to snowflake (#13286)
- ``KubernetesExecutor`` should accept images from ``executor_config`` (#13074)
- Scheduler should acknowledge active runs properly (#13803)
- Bugfix: Unable to import Airflow plugins on Python 3.8 (#12859)
- Include ``airflow/contrib/executors`` in the dist package
- Pin Click version for Python 2.7 users
- Ensure all statsd timers use millisecond values. (#10633)
- [``kubernetes_generate_dag_yaml``] - Fix dag yaml generate function (#13816)
- Fix ``airflow tasks clear` cli command with ``--yes`` (#14188)
- Fix permission error on non-POSIX filesystem (#13121) (#14383)
- Fixed deprecation message for "variables" command (#14457)
- BugFix: fix the ``delete_dag`` function of json_client (#14441)
- Fix merging of secrets and configmaps for ``KubernetesExecutor`` (#14090)
- Fix webserver exiting when gunicorn master crashes (#13470)
- Bump ``ini`` from 1.3.5 to 1.3.8 in ``airflow/www_rbac``
- Bump ``datatables.net`` from 1.10.21 to 1.10.23 in ``airflow/www_rbac``
- Webserver: Sanitize string passed to origin param (#14738)
- Make ``rbac_app``'s ``db.session`` use the same timezone with ``@provide_session`` (#14025)
Improvements
""""""""""""
- Adds airflow as viable docker command in official image (#12878)
- ``StreamLogWriter``: Provide (no-op) close method (#10885)
- Add 'airflow variables list' command for 1.10.x transition version (#14462)
Doc only changes
""""""""""""""""
- Update URL for Airflow docs (#13561)
- Clarifies version args for installing 1.10 in Docker (#12875)
Airflow 1.10.14, 2020-12-10
----------------------------
Bug Fixes
"""""""""
- BugFix: Tasks with ``depends_on_past`` or ``task_concurrency`` are stuck (#12663)
- Fix issue with empty Resources in executor_config (#12633)
- Fix: Deprecated config ``force_log_out_after`` was not used (#12661)
- Fix empty asctime field in JSON formatted logs (#10515)
- [AIRFLOW-2809] Fix security issue regarding Flask SECRET_KEY (#3651)
- [AIRFLOW-2884] Fix Flask SECRET_KEY security issue in www_rbac (#3729)
- [AIRFLOW-2886] Generate random Flask SECRET_KEY in default config (#3738)
- Add missing comma in setup.py (#12790)
- Bugfix: Unable to import Airflow plugins on Python 3.8 (#12859)
- Fix setup.py missing comma in ``setup_requires`` (#12880)
- Don't emit first_task_scheduling_delay metric for only-once dags (#12835)
Improvements
""""""""""""
- Update setup.py to get non-conflicting set of dependencies (#12636)
- Rename ``[scheduler] max_threads`` to ``[scheduler] parsing_processes`` (#12605)
- Add metric for scheduling delay between first run task & expected start time (#9544)
- Add new-style 2.0 command names for Airflow 1.10.x (#12725)
- Add Kubernetes cleanup-pods CLI command for Helm Chart (#11802)
- Don't let webserver run with dangerous config (#12747)
- Replace pkg_resources with ``importlib.metadata`` to avoid VersionConflict errors (#12694)
Doc only changes
""""""""""""""""
- Clarified information about supported Databases
Airflow 1.10.13, 2020-11-25
----------------------------
New Features
""""""""""""
- Add "already checked" to failed pods in K8sPodOperator (#11368)
- Pass SQLAlchemy engine options to FAB based UI (#11395)
- [AIRFLOW-4438] Add Gzip compression to S3_hook (#8571)
- Add permission "extra_links" for Viewer role and above (#10719)
- Add generate_yaml command to easily test KubernetesExecutor before deploying pods (#10677)
- Add Secrets backend for Microsoft Azure Key Vault (#10898)
Bug Fixes
"""""""""
- SkipMixin: Handle empty branches (#11120)
- [AIRFLOW-5274] dag loading duration metric name too long (#5890)
- Handle no Dagrun in DagrunIdDep (#8389) (#11343)
- Fix Kubernetes Executor logs for long dag names (#10942)
- Add on_kill support for the KubernetesPodOperator (#10666)
- KubernetesPodOperator template fix (#10963)
- Fix displaying of add serialized_dag table migration
- Fix Start Date tooltip on DAGs page (#10637)
- URL encode execution date in the Last Run link (#10595)
- Fixes issue with affinity backcompat in Airflow 1.10
- Fix KubernetesExecutor import in views.py
- Fix issues with Gantt View (#12419)
- Fix Entrypoint and _CMD config variables (#12411)
- Fix operator field update for SerializedBaseOperator (#10924)
- Limited cryptography to < 3.2 for Python 2.7
- Install cattr on Python 3.7 - Fix docs build on RTD (#12045)
- Limit version of marshmallow-sqlalchemy
- Pin `kubernetes` to a max version of 11.0.0 (#11974)
- Use snakebite-py3 for HDFS dependency for Python3 (#12340)
- Removes snakebite kerberos dependency (#10865)
- Fix failing dependencies for FAB and Celery (#10828)
- Fix pod_mutation_hook for 1.10.13 (#10850)
- Fix formatting of Host information
- Fix Logout Google Auth issue in Non-RBAC UI (#11890)
- Add missing imports to app.py (#10650)
- Show Generic Error for Charts & Query View in old UI (#12495)
- TimeSensor should respect the default_timezone config (#9699)
- TimeSensor should respect DAG timezone (#9882)
- Unify user session lifetime configuration (#11970)
- Handle outdated webserver session timeout gracefully. (#12332)
Improvements
""""""""""""
- Add XCom.deserialize_value to Airflow 1.10.13 (#12328)
- Mount airflow.cfg to pod_template_file (#12311)
- All k8s object must comply with JSON Schema (#12003)
- Validate Airflow chart values.yaml & values.schema.json (#11990)
- Pod template file uses custom custom env variable (#11480)
- Bump attrs and cattrs dependencies (#11969)
- Bump attrs to > 20.0 (#11799)
- [AIRFLOW-3607] Only query DB once per DAG run for TriggerRuleDep (#4751)
- Rename task with duplicate task_id
- Manage Flask AppBuilder Tables using Alembic Migrations (#12352)
- ``airflow test`` only works for tasks in 1.10, not whole dags (#11191)
- Improve warning messaging for duplicate task_ids in a DAG (#11126)
- Pins moto to 1.3.14 (#10986)
- DbApiHook: Support kwargs in get_pandas_df (#9730)
- Make grace_period_seconds option on K8sPodOperator (#10727)
- Fix syntax error in Dockerfile 'maintainer' Label (#10899)
- The entrypoints in Docker Image should be owned by Airflow (#10853)
- Make dockerfiles Google Shell Guide Compliant (#10734)
- clean-logs script for Dockerfile: trim logs before sleep (#10685)
- When sending tasks to celery from a sub-process, reset signal handlers (#11278)
- SkipMixin: Add missing session.commit() and test (#10421)
- Webserver: Further Sanitize values passed to origin param (#12459)
- Security upgrade lodash from 4.17.19 to 4.17.20 (#11095)
- Log instead of raise an Error for unregistered OperatorLinks (#11959)
- Mask Password in Log table when using the CLI (#11468)
- [AIRFLOW-3607] Optimize dep checking when depends on past set and concurrency limit
- Execute job cancel HTTPRequest in Dataproc Hook (#10361)
- Use rst lexer to format Airflow upgrade check output (#11259)
- Remove deprecation warning from contrib/kubernetes/pod.py
- adding body as templated field for CloudSqlImportOperator (#10510)
- Change log level for User's session to DEBUG (#12414)
Deprecations
""""""""""""
- Deprecate importing Hooks from plugin-created module (#12133)
- Deprecate adding Operators and Sensors via plugins (#12069)
Doc only changes
""""""""""""""""
- [Doc] Correct description for macro task_instance_key_str (#11062)
- Checks if all the libraries in setup.py are listed in installation.rst file (#12023)
- Revise "Project Focus" copy (#12011)
- Move Project focus and Principles higher in the README (#11973)
- Remove archived link from README.md (#11945)
- Update download url for Airflow Version (#11800)
- Add Project URLs for PyPI page (#11801)
- Move Backport Providers docs to our docsite (#11136)
- Refactor rebase copy (#11030)
- Add missing images for kubernetes executor docs (#11083)
- Fix indentation in executor_config example (#10467)
- Enhanced the Kubernetes Executor doc (#10433)
- Refactor content to a markdown table (#10863)
- Rename "Beyond the Horizon" section and refactor content (#10802)
- Refactor official source section to use bullets (#10801)
- Add section for official source code (#10678)
- Add redbubble link to Airflow merchandise (#10359)
- README Doc: Link to Airflow directory in ASF Directory (#11137)
- Fix the default value for VaultBackend's config_path (#12518)
Airflow 1.10.12, 2020-08-25
----------------------------
New Features
""""""""""""
- Add DateTimeSensor (#9697)
- Add AirflowClusterPolicyViolation support to Airflow local settings (#10282)
- Get Airflow configs with sensitive data from Secret Backends (#9645)
- [AIRFLOW-4734] Upsert functionality for PostgresHook.insert_rows() (#8625)
- Allow defining custom XCom class (#8560)
Bug Fixes
"""""""""
- Add pre 1.10.11 Kubernetes Paths back with Deprecation Warning (#10067)
- Fixes PodMutationHook for backwards compatibility (#9903)
- Fix bug in executor_config when defining resources (#9935)
- Respect DAG Serialization setting when running sync_perm (#10321)
- Show correct duration on graph view for running task (#8311) (#8675)
- Fix regression in SQLThresholdCheckOperator (#9312)
- [AIRFLOW-6931] Fixed migrations to find all dependencies for MSSQL (#9891)
- Avoid sharing session with RenderedTaskInstanceFields write and delete (#9993)
- Fix clear future recursive when ExternalTaskMarker is used (#9515)
- Handle IntegrityError while creating TIs (#10136)
- Fix airflow-webserver startup errors when using Kerberos Auth (#10047)
- Fixes treatment of open slots in scheduler (#9316) (#9505)
- Fix KubernetesPodOperator reattachment (#10230)
- Fix more PodMutationHook issues for backwards compatibility (#10084)
- [AIRFLOW-5391] Do not re-run skipped tasks when they are cleared (#7276)
- Fix task_instance_mutation_hook (#9910)
- Fixes failing formatting of DAG file containing {} in docstring (#9779)
- Fix is_terminal_support_colors function (#9734)
- Fix PythonVirtualenvOperator when using ``provide_context=True`` (#8256)
- Fix issue with mounting volumes from secrets (#10366)
- BugFix: K8s Executor Multinamespace mode is evaluated to true by default (#10410)
- Make KubernetesExecutor recognize kubernetes_labels (#10412)
- Fix broken Kubernetes PodRuntimeInfoEnv (#10478)
Improvements
""""""""""""
- Use Hash of Serialized DAG to determine DAG is changed or not (#10227)
- Update Serialized DAGs in Webserver when DAGs are Updated (#9851)
- Do not Update Serialized DAGs in DB if DAG did not change (#9850)
- Add __repr__ to SerializedDagModel (#9862)
- Update JS packages to latest versions (#9811) (#9921)
- UI Graph View: Focus upstream / downstream task dependencies on mouseover (#9303)
- Allow ``image`` in ``KubernetesPodOperator`` to be templated (#10068)
- [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod (#7523)
- Improve process terminating in scheduler_job (#8064)
- Replace deprecated base classes used in bigquery_check_operator (#10272)
- [AIRFLOW-5897] Allow setting -1 as pool slots value in webserver (#6550)
- Limit all google-cloud api to <2.0.0 (#10317)
- [AIRFLOW-6706] Lazy load operator extra links (#7327) (#10318)
- Add Snowflake support to SQL operator and sensor (#9843)
- Makes multi-namespace mode optional (#9570)
- Pin pyarrow < 1.0
- Pin pymongo version to <3.11.0
- Pin google-cloud-container to <2 (#9901)
- Dockerfile: Remove package.json and yarn.lock from the prod image (#9814)
- Dockerfile: The group of embedded DAGs should be root to be OpenShift compatible (#9794)
- Update upper limit of flask-swagger, gunicorn & jinja2 (#9684)
- Webserver: Sanitize values passed to origin param (#10334)
- Sort connection type list in add/edit page alphabetically (#8692)
Doc only changes
""""""""""""""""
- Add new committers: Ry Walker & Leah Cole to project.rst (#9892)
- Add Qingping Hou to committers list (#9725)
- Updated link to official documentation (#9629)
- Create a short-link for Airflow Slack Invites (#10034)
- Fix docstrings in BigQueryGetDataOperator (#10042)
- Set language on code-block on docs/howto/email-config.rst (#10238)
- Remove duplicate line from 1.10.10 CHANGELOG (#10289)
- Improve heading on Email Configuration page (#10175)
- Fix link for the Jinja Project in docs/tutorial.rst (#10245)
- Create separate section for Cron Presets (#10247)
- Add Syntax Highlights to code-blocks in docs/best-practices.rst (#10258)
- Fix docstrings in BigQueryGetDataOperator (#10042)
- Fix typo in Task Lifecycle section (#9867)
- Make Secret Backend docs clearer about Variable & Connection View (#8913)
Airflow 1.10.11, 2020-07-10
-----------------------------
New Features
""""""""""""
- Add task instance mutation hook (#8852)
- Allow changing Task States Colors (#9520)
- Add support for AWS Secrets Manager as Secrets Backend (#8186)
- Add Airflow info command to the CLI (#8704)
- Add Local Filesystem Secret Backend (#8596)
- Add Airflow config CLI command (#8694)
- Add Support for Python 3.8 (#8836)(#8823)
- Allow K8S worker pod to be configured from JSON/YAML file (#6230)
- Add quarterly to crontab presets (#6873)
- Add support for ephemeral storage on KubernetesPodOperator (#6337)
- Add AirflowFailException to fail without any retry (#7133)
- Add SQL Branch Operator (#8942)
Bug Fixes
"""""""""
- Use NULL as dag.description default value (#7593)
- BugFix: DAG trigger via UI error in RBAC UI (#8411)
- Fix logging issue when running tasks (#9363)
- Fix JSON encoding error in DockerOperator (#8287)
- Fix alembic crash due to typing import (#6547)
- Correctly restore upstream_task_ids when deserializing Operators (#8775)
- Correctly store non-default Nones in serialized tasks/dags (#8772)
- Correctly deserialize dagrun_timeout field on DAGs (#8735)
- Fix tree view if config contains " (#9250)
- Fix Dag Run UI execution date with timezone cannot be saved issue (#8902)
- Fix Migration for MSSQL (#8385)
- RBAC ui: Fix missing Y-axis labels with units in plots (#8252)
- RBAC ui: Fix missing task runs being rendered as circles instead (#8253)
- Fix: DagRuns page renders the state column with artifacts in old UI (#9612)
- Fix task and dag stats on home page (#8865)
- Fix the trigger_dag api in the case of nested subdags (#8081)
- UX Fix: Prevent undesired text selection with DAG title selection in Chrome (#8912)
- Fix connection add/edit for spark (#8685)
- Fix retries causing constraint violation on MySQL with DAG Serialization (#9336)
- [AIRFLOW-4472] Use json.dumps/loads for templating lineage data (#5253)
- Restrict google-cloud-texttospeech to <v2 (#9137)
- Fix pickling failure when spawning processes (#8671)
- Pin Version of azure-cosmos to <4 (#8956)
- Azure storage 0.37.0 is not installable any more (#8833)
- Fix modal_backdrop z-index in the UI (#7313)
- Fix Extra Links in Gantt View (#8308)
- Bug fix for EmrAddStepOperator init with cluster_name error (#9235)
- Fix KubernetesPodOperator pod name length validation (#8829)
- Fix non updating DAG code by checking against last modification time (#8266)
- BugFix: Unpausing a DAG with catchup=False creates an extra DAG run (#8776)
Improvements
""""""""""""
- Improve add_dag_code_table migration (#8176)
- Persistent display/filtering of DAG status (#8106)
- Set unique logger names (#7330)
- Update the version of cattrs from 0.9 to 1.0 to support Python 3.8 (#7100)
- Reduce response payload size of /dag_stats and /task_stats (#8655)
- Add authenticator parameter to snowflake_hook (#8642)
- Show "Task Reschedule" table in Airflow Webserver (#9521)
- Change worker_refresh_interval fallback to default of 30 (#9588)
- Use pformat instead of str to render arguments in WebUI (#9587)
- Simplify DagFileProcessorManager (#7521)
- Reload gunicorn when plugins has been changed (#8997)
- Fix the default value for store_dag_code (#9554)
- Add support for fetching logs from running pods (#8626)
- Persist start/end date and duration for DummyOperator Task Instance (#8663)
- Ensure "started"/"ended" in tooltips are not shown if job not started (#8667)
- Add context to execution_date_fn in ExternalTaskSensor (#8702)
- Avoid color info in response of ``/dag_stats`` & ``/task_stats`` (#8742)
- Make loading plugins from entrypoint fault-tolerant (#8732)
- Refactor Kubernetes worker config (#7114)
- Add default ``conf`` parameter to Spark JDBC Hook (#8787)
- Allow passing backend_kwargs to AWS SSM client (#8802)
- Filter dags by clicking on tag (#8897)
- Support k8s auth method in Vault Secrets provider (#8640)
- Monitor pods by labels instead of names (#6377)
- Optimize count query on /home (#8729)
- Fix JSON string escape in tree view (#8551)
- Add TaskInstance state to TI Tooltip to be colour-blind friendlier (#8910)
- Add a tip to trigger DAG screen (#9049)
- Use Markup for htmlcontent for landing_times (#9242)
- Pinning max pandas version to 2.0 (lesser than) to allow pandas 1.0 (#7954)
- Update example webserver_config.py to show correct CSRF config (#8944)
- Fix displaying Executor Class Name in "Base Job" table (#8679)
- Use existing DagBag for 'dag_details' & 'trigger' Endpoints (#8501)
- Flush pending Sentry exceptions before exiting (#7232)
- Display DAG run conf in the list view (#6794)
- Fix performance degradation when updating dagrun state (#8435)
- Don't use the ``|safe`` filter in code, it's risky (#9180)
- Validate only task commands are run by executors (#9178)
- Show Deprecation warning on duplicate Task ids (#8728)
- Move DAG._schedule_interval logic out of ``DAG.__init__`` (#8225)
- Make retrieving Paused Dag ids a separate method (#7587)
- Bulk fetch paused_dag_ids (#7476)
- Add a configurable DAGs volume mount path for Kubernetes (#8147)
- Add schedulername option for KubernetesPodOperator (#6088)
- Support running git sync container as root (#6312)
- Add extra options for Slack Webhook operator and Slack hook (#9409)
- Monkey patch greenlet Celery pools (#8559)
- Decrypt secrets from SystemsManagerParameterStoreBackend (#9214)
- Prevent clickable sorting on non sortable columns in TI view (#8681)
- Make hive macros py3 compatible (#8598)
- Fix SVG tooltip positioning with custom scripting (#8269)
- Avoid unnecessary sleep to maintain local task job heart rate (#6553)
- Include some missing RBAC roles on User and Viewer roles (#9133)
- Show Dag's Markdown docs on Tree View (#9448)
- Improved compatibility with Python 3.5+ - Convert signal.SIGTERM to int (#9207)
- Add 'main' param to template_fields in DataprocSubmitPySparkJobOperator (#9154)
- Make it possible to silence warnings from Airflow (#9208)
- Remove redundant count query in BaseOperator.clear() (#9362)
- Fix DB migration message (#8988)
- Fix awkward log info in dbapi_hook (#8482)
- Fix Celery default to no longer allow pickle (#7205)
- Further validation that only task commands are run by executors (#9240)
- Remove vendored nvd3 and slugify libraries (#9136)
- Enable configurable git sync depth (#9094)
- Reduce the required resources for the Kubernetes's sidecar (#6062)
- Refactor K8S codebase with k8s API models (#5481)
- Move k8s executor out of contrib to closer match master (#8904)
- Allow filtering using "event" and "owner" in "Log" view (#4881)
- Add Yandex.Cloud custom connection to 1.10 (#8791)
- Add table-hover css class to DAGs table (#5033)
- Show un/pause errors in dags view. (#7669)
- Restructure database queries on /home (#4872)
- Add Cross Site Scripting defense (#6913)
- Make Gantt tooltip the same as Tree and Graph view (#8220)
- Add config to only delete worker pod on task failure (#7507)(#8312)
- Remove duplicate error message on chart connection failure (#8476)
- Remove default value spark_binary (#8508)
- Expose Airflow Webserver Port in Production Docker Image (#8228)
- Commit after each alembic migration (#4797)
- KubernetesPodOperator fixes and test (#6524)
- Docker Image: Add ADDITIONAL_AIRFLOW_EXTRAS (#9032)
- Docker Image: Add ADDITIONAL_PYTHON_DEPS (#9031)
- Remove httplib2 from Google requirements (#9194)
- Merging multiple SQL operators (#9124)
- Adds hive as extra in pyhive dependency (#9075)
- Change default auth for experimental backend to deny_all (#9611)
- Restrict changing XCom values from the Webserver (#9614)
- Add __repr__ for DagTag so tags display properly in /dagmodel/show (#8719)
- Functionality to shuffle HMS connections used by HiveMetastoreHook facilitating load balancing (#9280)
- Expose SQLAlchemy's connect_args and make it configurable (#6478)
Doc only changes
""""""""""""""""
- Add docs on using DAGRun.conf (#9578)
- Enforce code-block directives in doc (#9443)
- Carefully parse warning messages when building documentation (#8693)
- Make KubernetesPodOperator clear in docs (#8444)
- Improve language in Pod Mutation Hook docs (#8445)
- Fix formatting of Pool docs in concepts.rst (#8443)
- Make doc clearer about Airflow Variables using Environment Variables (#8427)
- Fix pools doc for LocalExecutor (#7643)
- Add Local and Sequential Executors to Doc (#8084)
- Add documentation for CLI command Airflow dags test (#8251)
- Fix typo in DAG Serialization documentation (#8317)
- Add scheduler in production section (#7351)
- Add a structural dag validation example (#6727)
- Adding Task re-run documentation (#6295)
- Fix outdated doc on settings.policy (#7532)
- Add docs about reload_on_plugin_change option (#9575)
- Add copy button to Code Blocks in Airflow Docs (#9450)
- Update commands in docs for v1.10+ (#9585)
- Add more info on dry-run CLI option (#9582)
- Document default timeout value for SSHOperator (#8744)
- Fix docs on creating CustomOperator (#8678)
- Enhanced documentation around Cluster Policy (#8661)
- Use sphinx syntax in concepts.rst (#7729)
- Update README to remove Python 3.8 limitation for Master (#9451)
- Add note about using dag_run.conf in BashOperator (#9143)
- Improve tutorial - Include all imports statements (#8670)
- Added more precise Python requirements to README.md (#8455)
- Fix Airflow Stable version in README.md (#9360)
- Update AWS connection example to show how to set from env var (#9191)
- Fix list formatting of plugins doc. (#8873)
- Add 'Version Added' on Secrets Backend docs (#8264)
- Simplify language re roll-your-own secrets backend (#8257)
- Add installation description for repeatable PyPi installation (#8513)
- Add note extra links only render on when using RBAC webserver (#8788)
- Remove unused Airflow import from docs (#9274)
- Add PR/issue note in Contribution Workflow Example (#9177)
- Don't use the term "whitelist" - language matters (#9174)
- Add docs to change Colors on the Webserver (#9607)
- Change 'initiate' to 'initialize' in installation.rst (#9619)
- Replace old Variables View Screenshot with new (#9620)
- Replace old SubDag zoom screenshot with new (#9621)
- Update docs about the change to default auth for experimental API (#9617)
Airflow 1.10.10, 2020-04-09
---------------------------
New Features
""""""""""""
- [AIRFLOW-7048] Allow user to chose timezone to use in UI (#8046)
- Add Production Docker image support (#7832)
- Get Airflow Variables from Environment Variables (#7923)
- Get Airflow Variables from Hashicorp Vault (#7944)
- Get Airflow Variables from AWS Systems Manager Parameter Store (#7945)
- Get Airflow Variables from GCP Secrets Manager (#7946)
- [AIRFLOW-5705] Add secrets backend and support for AWS SSM / Get Airflow Connections from AWS Parameter Store(#6376)
- [AIRFLOW-7104] Add Secret backend for GCP Secrets Manager / Get Airflow Connections from GCP Secrets Manager (#7795)
- [AIRFLOW-7076] Add support for HashiCorp Vault as Secrets Backend / Get Airflow Connections from Hashicorp Vault (#7741)
- [AIRFLOW-6685] Add ThresholdCheckOperator (#7353)
- [AIRFLOW-7080] Add API endpoint to return a DAG's paused state (#7737)
Bug Fixes
"""""""""
- BugFix: Show task_id in the Graph View tooltip (#7859)
- [AIRFLOW-6730] Use total_seconds instead of seconds (#7363)
- [AIRFLOW-6167] Escape column name in create table in hive (#6741)
- [AIRFLOW-6628] DAG auto-complete now suggests from all accessible DAGs (#7251)
- [AIRFLOW-7113] Fix gantt render error (#7913)
- [AIRFLOW-6399] Add _access control to validate deserialized DAGs (#7896)
- [AIRFLOW-6399] Serialization: DAG access_control field should be decorated field in DAG serialization (#7879)
- [AIRFLOW-4453] Make behavior of `none_failed` consistent with documentation (#7464)
- [AIRFLOW-4363] Fix JSON encoding error (#7628)
- [AIRFLOW-6683] Run REST API tests when DAGs are serialized (#7352)
- [AIRFLOW-6704] Copy common TaskInstance attributes from Task (#7324)
- [AIRFLOW-6734] Use configured base_template instead of hard-coding (#7367)