Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support limiting deltas entries in a compaction round #69

Open
raghumdani opened this issue Feb 4, 2023 · 2 comments
Open

Support limiting deltas entries in a compaction round #69

raghumdani opened this issue Feb 4, 2023 · 2 comments
Assignees
Labels
P3 Resolve if not working on P0/P1/P2 (< 8 weeks)

Comments

@raghumdani
Copy link
Collaborator

Currently, we only limit deltas in a compaction round based on total object store memory available in a cluster. When there is a very large delta that contains many manifest files, we still have to limit them and perform re-batching.

@raghumdani raghumdani self-assigned this Feb 4, 2023
@pdames pdames added P1 Resolve if not working on P0 (< 2 weeks) P0 Resolve ASAP (< 1 week) and removed P1 Resolve if not working on P0 (< 2 weeks) P0 Resolve ASAP (< 1 week) labels Feb 8, 2023
@pdames
Copy link
Member

pdames commented Feb 9, 2023

From #70:

... the current contract of compaction assumes that each round must be able to compact at least one delta. To work with extremely large deltas we'll need to drive that down to at least file-level granularity (which will drive subsequent changes into the Round Completion File and each round that reads it to determine a starting point). Future improvements would then include driving each round down to record-level granularity to work with files that are too large to complete in a single round.

@raghumdani
Copy link
Collaborator Author

Primary key index building is a pre-requisite to running multiple rounds: #63

@pdames pdames added this to the Compaction with PKI milestone Feb 21, 2023
@pdames pdames added P3 Resolve if not working on P0/P1/P2 (< 8 weeks) and removed P1 Resolve if not working on P0 (< 2 weeks) labels Feb 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P3 Resolve if not working on P0/P1/P2 (< 8 weeks)
Projects
None yet
Development

No branches or pull requests

2 participants