Pagination to Jetstream KV

[m13t]

Proposed change

It would be nice to have pagination built in to the Jetstream KV operations. I know internally, I could load the stream and fetch a batch with a given size and start sequence to achieve this, but it would be a lot cleaner if this was supported in the KV api to match operations supported by traditional KV stores.

As this functionality is technically there for streams already, it's more a case of adding syntactical sugar to the API rather than having to create this functionality from scratch.

Use case

Whilst it is currently possible to get a list of keys in a KV bucket already, this is channel based with no option to specify a starting sequence as this is itself just sugar for the watcher implementation (which also does not support start sequence).

In situations where it is not desirable or feasible to load all keys into memory and paginate from the in-memory list, it would be very useful to be able to simply retrieve a list of keys of a given size starting with an optional sequence number. This way state would not need to be stored within an application that is batch iterating over keys in a bucket, such as perhaps a web-based application that's displaying a list of keys in a given bucket.

Contribution

I'm not overly familiar with the codebase beyond the exposed APIs but I would be happy to attempt the implementation. However, I did note that the internal stream sequence number is not exposed on values retrieved from a KV bucket either via Get or from KeyValueEntry interface.

[ripienaar]

Nats 2.11 introduces batching on the direct get method we use to get data from streams for KV so should become easier to add.

We have a few more improvements specifically for KV coming in 2.11 so clients could take a bit to get it all but it’s on the cards for sure.