Implement a pool of topic-partition workers

[k32]

group_subscriber_v2 behavior adds parallelism to group subscribers. However, spawning too many parallel workers may become a problem. There should be a way to put an upper limit on the number of parallel workers.

[jeffrade]

However, spawning too many parallel workers may become a problem.

Would having 6 partitions cause disruption? How many is too many? I'm seeing connection issues to Confluent.

[k32]

Hi!

How many is too many?

It depends on the broker settings. We don't use Confluent-managed Kafka, so I don't know how many parallel connections per host they allow. In relative terms 6 doesn't seem too many, we easily use 20 per host.

[jeffrade]

@k32 Thanks, appreciate the quick response.

Any idea(s) what might be causing our issue here: #357