Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compaction throughput degraded after 1.8 -> 1.9.1 upgrade #3817

Open
kolesnikovae opened this issue Jan 6, 2025 · 0 comments
Open

Compaction throughput degraded after 1.8 -> 1.9.1 upgrade #3817

kolesnikovae opened this issue Jan 6, 2025 · 0 comments
Assignees
Labels
storage Low level storage matters type/bug Something isn't working

Comments

@kolesnikovae
Copy link
Collaborator

From https://grafana.slack.com/archives/C049PLMV8TB/p1735948533932149:

After upgrading from Pyroscope 1.8.0 -> 1.9.1, we observed a massive decrease in throughput of compaction. (Behavior persists in 1.10 and 1.11.) After the upgrade, we've observed a compactor taking ~12 hours to compact/write a ~12 GB block. So the compactor throughput is <<1 MB/s. All the while the compactor process is using 100% of the CPU allocated to it. This represents a >>10x slowdown versus 1.8. (Before the upgrade, we were able to perform 3 rounds of compaction (level 4) from 1 compactor process. After, we had to 10x the compactor CPU footprint just to perform 1 round of compaction. Attempting an additional level of compaction beyond 1 hour ranges just wasn't viable since it took too much CPU resources.)

Also, I downgraded an instance to 1.8.0 hoping to see the old/fast performance come back. But the compactor flamegraph looks very similar as 1.11.0. I'm really confused because our time series metrics from Pyroscope clearly show the compactor performance degrading after the 1.8.0 -> 1.9.1 upgrade. So I'm at a loss here.

@kolesnikovae kolesnikovae added storage Low level storage matters type/bug Something isn't working labels Jan 6, 2025
@kolesnikovae kolesnikovae self-assigned this Jan 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
storage Low level storage matters type/bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant