You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, we only limit deltas in a compaction round based on total object store memory available in a cluster. When there is a very large delta that contains many manifest files, we still have to limit them and perform re-batching.
The text was updated successfully, but these errors were encountered:
pdames
added
P1
Resolve if not working on P0 (< 2 weeks)
P0
Resolve ASAP (< 1 week)
and removed
P1
Resolve if not working on P0 (< 2 weeks)
P0
Resolve ASAP (< 1 week)
labels
Feb 8, 2023
... the current contract of compaction assumes that each round must be able to compact at least one delta. To work with extremely large deltas we'll need to drive that down to at least file-level granularity (which will drive subsequent changes into the Round Completion File and each round that reads it to determine a starting point). Future improvements would then include driving each round down to record-level granularity to work with files that are too large to complete in a single round.
Currently, we only limit deltas in a compaction round based on total object store memory available in a cluster. When there is a very large delta that contains many manifest files, we still have to limit them and perform re-batching.
The text was updated successfully, but these errors were encountered: