Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: new configuration parameters for input: kafka_franz #2921

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

darkxeno
Copy link

@darkxeno darkxeno commented Oct 8, 2024

This PR enables the following configuration parameters to be used for the input kafka_franz:

session_timeout: "Sets how long a member in the group can go between heartbeats. If a member does not heartbeat in this timeout, the broker will remove the member from the group and initiate a rebalance."
defaults to 45 secs

heartbeat_period "Sets how long a group member goes between heartbeats to Kafka. Kafka uses heartbeats to ensure that a group member's session stays active. This value can be any value lower than the session timeout, but should be no higher than 1/3rd the session timeout."
defaults to 3 secs

rebalance_timeout "Sets how long group members are allowed to take when a rebalance has begun. This timeout is how long all members are allowed to complete work and commit offsets, minus the time it took to detect the rebalance (from a heartbeat)."
defaults to 60 secs

Default values are based on frank-go defaults:
https://github.com/twmb/franz-go/blob/b77dd13e2bfaee7f5181df27b40ee4a4f6a73b09/pkg/kgo/config.go#L1448
https://github.com/twmb/franz-go/blob/b77dd13e2bfaee7f5181df27b40ee4a4f6a73b09/pkg/kgo/config.go#L1467
https://github.com/twmb/franz-go/blob/b77dd13e2bfaee7f5181df27b40ee4a4f6a73b09/pkg/kgo/config.go#L1481

Supporting these parameters helps to support workloads where processing 1 single message might take longer than 1 or 2 minutes, as longer session timeouts with longer heartbeat intervals helps to avoid or reduce rebalances.

@CLAassistant
Copy link

CLAassistant commented Oct 8, 2024

CLA assistant check
All committers have signed the CLA.

@darkxeno darkxeno changed the title Adding new configuration parameters to input: kafka_franz feat: new configuration parameters to input: kafka_franz Oct 8, 2024
@darkxeno darkxeno changed the title feat: new configuration parameters to input: kafka_franz feat: new configuration parameters for input: kafka_franz Oct 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants