compare() returns fewer exemplars than expected

[mapno]

Context

Exemplars were added to TraceQL metrics in #3824. Queries such as {} | rate(), {} | quantile_over_time(duration, 0,99), {} | compare({status=error}) return exemplars if enabled.

Exemplars in compare() are only returned for the series __meta_type="selection_total" and __meta_type="baseline_total", since every other series is a subset of the totals, and exemplars carry all labels of the query.

Problem

Queries that use compare() are inconsistent in returning exemplars—usually return very few exemplars and only in some series, instead of in all series.

There are a couple of potential issues at hand here:

• Sampling is very naive and the CallbackPredicate might be stopping collection too aggressively. The callback might be quitting before enough exemplars are collected.
• Exemplars might not be propagated correctly to all series. There are multiple places where series are merged, and exemplars could be lost there. The first place is in the engine itself. But there are followup calls in the queriers, generators and frontends.