support fallback configuration for KEDA autoscaling

[hobbsh]

What this PR does

Supports a fallback configuration for the KEDA autoscaling configuration in the mimir-distributed helm chart, so if/when the metrics endpoint being used to scale becomes unavailable, the ScaledObject will fallback to the configured replica count.

Which issue(s) this PR fixes or relates to

n/a

Checklist

• Tests updated.
• Documentation added.
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
• about-versioning.md updated with experimental features.

[CLAassistant]

All committers have signed the CLA.

[jhesketh]

Thank you for the contribution!

This will need an entry in operations/helm/charts/mimir-distributed/CHANGELOG.md.

I'm also curious on your opinion if a fallback if preferable to just maintaining the current number of replicas given that a deployment is also protected by the minReplicas? Would it not be better to maintain the current replicas and alert when KEDA isn't able to get the metrics? Or is the intention to generally have a fallback end up scaling up a deployment?

[hobbsh]

First off, thank you for the engagement!

Would it not be better to maintain the current replicas and alert when KEDA isn't able to get the metrics?

In my experience, if the distributors aren't scaling, they will quickly OOM and then we get into a catch 22 where we need to scale but can't because metrics can't be retrieved. KEDA created fallback for exactly this reason so I would much prefer to use it rather than try to rely on minReplicas, because for one it would lead to additional overhead/cost to do it that way. We use this for some internal services and it works well, so it would be great to have this option for Mimir. So yes, the intention is to have fallback scale up the deployment if we have a metrics blip. Obviously, it would be great to have the autoscaling metrics coming from an external source, but we are quite resource strapped and that's not the easiest option.

This will need an entry in operations/helm/charts/mimir-distributed/CHANGELOG.md

Done, thanks!