Reduce alerts due to temporary compaction #2497
Unanswered
martin-aders
asked this question in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Whenever max disk retention size is reached, a compaction is triggered. The compaction requires additional space temporarily until disk space may be freed. Having this alert fire "for: 1m" may cause false alerts during this compaction period unless you reserve a larger amount of disk size just for the short period when compaction occurs. On a real scenario, a "for: 10m" would be enough to give time to get disk space freed. May be a matter of taste, though. There may be alternative ways to increasing the "for" time too, to cover systems with slower disks.
Log compaction example, showing percentage of free disk from ~40% down to almost 0%:
Code reference: https://github.com/prometheus-operator/kube-prometheus/blob/d04c1e140ad406cd14c0040a2e891d389b69814b/manifests/kubernetesControlPlane-prometheusRule.yaml#L380C1-L380C14
Documentation: https://prometheus.io/docs/prometheus/latest/storage/#compaction
Beta Was this translation helpful? Give feedback.
All reactions