Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support table compaction source rebases and refactor compact_partition #70

Merged
merged 3 commits into from
Feb 10, 2023
Merged

Support table compaction source rebases and refactor compact_partition #70

merged 3 commits into from
Feb 10, 2023

Conversation

pdames
Copy link
Member

@pdames pdames commented Feb 4, 2023

See user guide in included README.md for compaction source rebase use-cases and invocation examples.
As part of this PR, the compact_partition API has also been refactored for better backwards-compatibility going forward by using keyword arguments for all optional args.

@pdames
Support table compaction source rebases and refactor compact_partitio…
08fb6bc 
…n API to start using backwards-compatible keyword args.
@pdames pdames changed the title Support table compaction source rebases, refactor compact_partition Support table compaction source rebases and refactor compact_partition Feb 4, 2023
@pdames pdames linked an issue Feb 4, 2023 that may be closed by this pull request
Support Primary Key Index Backfill From Existing Compacted Table #67
Closed
rkenmi
rkenmi reviewed Feb 6, 2023
View reviewed changes
deltacat/compute/compactor/README.md Outdated Show resolved Hide resolved
deltacat/compute/compactor/README.md Show resolved Hide resolved
raghumdani
raghumdani reviewed Feb 7, 2023
View reviewed changes
Copy link
Collaborator

@raghumdani raghumdani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good.

@valiantljk
Copy link
Collaborator

LGTM, with this pr, we also need to modify the io.discovery_deltas to combine both rebased and source tables' deltas in case there is no primary key index from prior round. #76

This was referenced Feb 9, 2023
Open
Open
@pdames
Copy link
Member Author

pdames commented Feb 10, 2023

LGTM, with this pr, we also need to modify the io.discovery_deltas to combine both rebased and source tables' deltas in case there is no primary key index from prior round. #76

Will address this in a follow-up PR that branches compactor code into implementations that run w/ and w/o PKI.

raghumdani
raghumdani approved these changes Feb 10, 2023
View reviewed changes
Copy link
Collaborator

@raghumdani raghumdani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@pdames pdames merged commit 3ee0910 into ray-project:main Feb 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yiqinzhu yiqinzhu yiqinzhu left review comments

@rkenmi rkenmi rkenmi left review comments

@raghumdani raghumdani raghumdani approved these changes

@Zyiqin-Miranda Zyiqin-Miranda Awaiting requested review from Zyiqin-Miranda

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support Primary Key Index Backfill From Existing Compacted Table
5 participants
@pdames @valiantljk @rkenmi @raghumdani @yiqinzhu