Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade daft and adding two more tests for schema evolution usecase #337

Merged
merged 4 commits into from
Jul 19, 2024

Conversation

raghumdani
Copy link
Collaborator

  • Added couple of tests.
  • Upgraded daft version
  • This fixes an issue when column name not in parquet schema is passed.

Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 2.

Benchmark suite Current: 932e39d Previous: d77a208 Ratio
deltacat/tests/compute/test_compact_partition_incremental.py::test_compact_partition_incremental[1-incremental-pkstr-sknone-norcf_V1] 1.0810036052605032 iter/sec (stddev: 0) 2.2521535582165826 iter/sec (stddev: 0) 2.08
deltacat/tests/compute/test_compact_partition_incremental.py::test_compact_partition_incremental[1-incremental-pkstr-sknone-norcf_V2] 1.073704052871053 iter/sec (stddev: 0) 2.264678604965184 iter/sec (stddev: 0) 2.11
deltacat/tests/compute/test_compact_partition_incremental.py::test_compact_partition_incremental[2-incremental-pkstr-skstr-norcf_V1] 1.060370284060169 iter/sec (stddev: 0) 2.2689989361845813 iter/sec (stddev: 0) 2.14
deltacat/tests/compute/test_compact_partition_incremental.py::test_compact_partition_incremental[2-incremental-pkstr-skstr-norcf_V2] 1.067061887040253 iter/sec (stddev: 0) 2.266440743018205 iter/sec (stddev: 0) 2.12
deltacat/tests/compute/test_compact_partition_incremental.py::test_compact_partition_incremental[3-incremental-pkstr-multiskstr-norcf_V1] 1.0861229707623132 iter/sec (stddev: 0) 2.18698162316299 iter/sec (stddev: 0) 2.01
deltacat/tests/compute/test_compact_partition_incremental.py::test_compact_partition_incremental[3-incremental-pkstr-multiskstr-norcf_V2] 1.0873349355706126 iter/sec (stddev: 0) 2.2595342936230303 iter/sec (stddev: 0) 2.08
deltacat/tests/compute/test_compact_partition_incremental.py::test_compact_partition_incremental[5-incremental-decimal-pk-simple_V1] 1.0849034327942895 iter/sec (stddev: 0) 2.253116921393015 iter/sec (stddev: 0) 2.08
deltacat/tests/compute/test_compact_partition_incremental.py::test_compact_partition_incremental[5-incremental-decimal-pk-simple_V2] 0.9345476087354807 iter/sec (stddev: 0) 2.2460476215775427 iter/sec (stddev: 0) 2.40
deltacat/tests/compute/test_compact_partition_incremental.py::test_compact_partition_incremental[6-incremental-integer-pk-simple_V1] 1.0529803093998353 iter/sec (stddev: 0) 2.2710662045823544 iter/sec (stddev: 0) 2.16
deltacat/tests/compute/test_compact_partition_incremental.py::test_compact_partition_incremental[7-incremental-timestamp-pk-simple_V1] 1.091837877291071 iter/sec (stddev: 0) 2.2212402859331584 iter/sec (stddev: 0) 2.03
deltacat/tests/compute/test_compact_partition_incremental.py::test_compact_partition_incremental[7-incremental-timestamp-pk-simple_V2] 1.0553010952581514 iter/sec (stddev: 0) 2.1956702695295776 iter/sec (stddev: 0) 2.08
deltacat/tests/compute/test_compact_partition_incremental.py::test_compact_partition_incremental[8-incremental-decimal-timestamp-pk-multi_V1] 1.0384340841016202 iter/sec (stddev: 0) 2.264246475800043 iter/sec (stddev: 0) 2.18
deltacat/tests/compute/test_compact_partition_incremental.py::test_compact_partition_incremental[8-incremental-decimal-timestamp-pk-multi_V2] 1.0439911837241 iter/sec (stddev: 0) 2.215446181611629 iter/sec (stddev: 0) 2.12
deltacat/tests/compute/test_compact_partition_incremental.py::test_compact_partition_incremental[9-incremental-decimal-pk-multi-dup_V1] 1.0728523039100528 iter/sec (stddev: 0) 2.294583661883228 iter/sec (stddev: 0) 2.14
deltacat/tests/compute/test_compact_partition_incremental.py::test_compact_partition_incremental[10-incremental-decimal-pk-partitionless_V1] 1.02521479495587 iter/sec (stddev: 0) 2.2115377574614943 iter/sec (stddev: 0) 2.16
deltacat/tests/compute/test_compact_partition_incremental.py::test_compact_partition_incremental[10-incremental-decimal-pk-partitionless_V2] 1.076874352334765 iter/sec (stddev: 0) 2.253822151056081 iter/sec (stddev: 0) 2.09
deltacat/tests/compute/test_compact_partition_incremental.py::test_compact_partition_incremental[11-incremental-decimal-hash-bucket-single_V1] 0.8820309461810104 iter/sec (stddev: 0) 2.2273418119238033 iter/sec (stddev: 0) 2.53
deltacat/tests/compute/test_compact_partition_incremental.py::test_compact_partition_incremental[11-incremental-decimal-hash-bucket-single_V2] 1.0530298370486406 iter/sec (stddev: 0) 2.2643017544376827 iter/sec (stddev: 0) 2.15
deltacat/tests/compute/test_compact_partition_incremental.py::test_compact_partition_incremental[12-incremental-decimal-single-hash-bucket_V1] 1.029124782968862 iter/sec (stddev: 0) 2.297777655168341 iter/sec (stddev: 0) 2.23
deltacat/tests/compute/test_compact_partition_incremental.py::test_compact_partition_incremental[12-incremental-decimal-single-hash-bucket_V2] 1.1820900091113142 iter/sec (stddev: 0) 2.6295975732771018 iter/sec (stddev: 0) 2.22

This comment was automatically generated by workflow using github-action-benchmark.

@raghumdani raghumdani merged commit a2aadb9 into main Jul 19, 2024
3 checks passed
@raghumdani raghumdani deleted the daft-fix branch July 19, 2024 17:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants