You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
After upgrading from Pyroscope 1.8.0 -> 1.9.1, we observed a massive decrease in throughput of compaction. (Behavior persists in 1.10 and 1.11.) After the upgrade, we've observed a compactor taking ~12 hours to compact/write a ~12 GB block. So the compactor throughput is <<1 MB/s. All the while the compactor process is using 100% of the CPU allocated to it. This represents a >>10x slowdown versus 1.8. (Before the upgrade, we were able to perform 3 rounds of compaction (level 4) from 1 compactor process. After, we had to 10x the compactor CPU footprint just to perform 1 round of compaction. Attempting an additional level of compaction beyond 1 hour ranges just wasn't viable since it took too much CPU resources.)
Also, I downgraded an instance to 1.8.0 hoping to see the old/fast performance come back. But the compactor flamegraph looks very similar as 1.11.0. I'm really confused because our time series metrics from Pyroscope clearly show the compactor performance degrading after the 1.8.0 -> 1.9.1 upgrade. So I'm at a loss here.
The text was updated successfully, but these errors were encountered:
From https://grafana.slack.com/archives/C049PLMV8TB/p1735948533932149:
The text was updated successfully, but these errors were encountered: