Lagging exporters in benchmarks.

[rodrigo-lourenco-lopes]

The exporters seem to be lagging behind quite a lot and exporting at a slower pace than what we expect them to do, for example, capping at around 200 to 300 while processing is at 2000 in normal benchmarks.

This has been observed in many weekly benchmarks and needs to be investigated as it can possibly be a regression.

Seems to be affecting mostly the standard benchmarks and not the realistic ones. (It does not seem to affect the mixed ones as these process and export way bellow 200 records per second, so its harder to judge)

This images show a recently created benchmark that is processing at around 2k and exporting between 200 and 300, while building up the backlog of records not exported.

Seems easy to replicate just by running a benchmark with a significantly high load to observe this.

[aabouzaid]

@rodrigo-lourenco-lopes Thanks for bringing up this topic 🙌

I just want to confirm if those images were before or after the resource requests and limits adjustment, which has been done in #2702.

[rodrigo-lourenco-lopes]

Our latest benchmarks are also down, where the backlog seems to have filled quite quickly, so I think it is still happening, although the examples that I had investigated were before those changes.