Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[INLONG-11386][TubeMQ] Use local files to save consumer group offset information #11395

Merged
merged 2 commits into from
Oct 23, 2024

Conversation

gosonzhang
Copy link
Contributor

Fixes #11386

image

Optimize content:

  1. Broker regularly refreshes the consumer group offset information in memory to the file Offset Cache according to the configured offsetStgCacheFlushMs value;
  2. The consumer group offset information stored in the file Offset Cache is regularly written to the offsetStgFilePath/offsetDir directory according to the configured offsetStgFileSyncMs value. The file name successfully saved is offsets.meta. The persistence logic is as follows:
  • Persistently save the consumer group offset information to the file offsets.tmp. If the write fails, output the failure log, alarm and exit the persistence process;
  • Read the offsets.tmp file and compare it with the previously written information to see if it is consistent. If it is inconsistent, output the failure log and exit the persistence process;
  • Migrate the file from offsets.tmp to offsets.mid. If the offsets.meta file currently exists, change offsets.meta to offsets.old;
  • Migrate the file from offsets.mid to offsets.meta, and delete the offsets.old file at the same time.
  1. Broker supports saving the current consumer group's Offset collection information to the backupPath path specified in the API interface through the HTTP API admin_backup_group_offsets. The saved file name is offsets.meta.

@gosonzhang gosonzhang merged commit 1272609 into apache:master Oct 23, 2024
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Improve][TubeMQ] Use local files to save consumer group offset information
3 participants