Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to bootstrap messages based on message type #13

Open
whynick1 opened this issue Jul 8, 2019 · 0 comments
Open

Add ability to bootstrap messages based on message type #13

whynick1 opened this issue Jul 8, 2019 · 0 comments

Comments

@whynick1
Copy link
Contributor

whynick1 commented Jul 8, 2019

When a Waltz log is shared amongst several applications, it might be useful for some applications to ignore messages that they don't care about. One way to ignore irrelevant message types would be to do server-side filtering, where the Waltz server or storage node drops unwanted messages. This is useful in a case where the application cares about messages across the whole timeline of the log.

A second use case is when the application is cares about only a subset of messages that might or might not be distributed all the way back to the beginning of the log. In such a case, it would be useful to have the application request the lowest watermark for the union of all the message types, so that the application can start reading from that point.

For example, suppose we have a Waltz log that's 2 years old. Four months ago, a new message type was added. A new application is being written that pays attention only to the new events. Having it fetch the low watermark for the new message type means that it can start reading from exactly the point in the log where the first message type first occurs, and allows the application to skip the vast majority of the irrelevant messages.

Note that this will have to be per-partition.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant