feat(nns): Add pagination to list_neurons API #3358
Conversation
max-dfinity
force-pushed
the
msum/list-neurons-pagination
branch
4 times, most recently
from
9f83272 to
a654324
Compare
max-dfinity
force-pushed
the
msum/list-neurons-pagination
branch
from
a654324 to
2d22844
Compare
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
If this pull request affects the behavior of any canister owned by
the Governance team, remember to update the corresponding
unreleased_changes.md file(s).
To acknowldge this reminder (and unblock the PR), dismiss this
code review by going to the bottom of the pull request page, and
supply one of the following reasons:
-
Done.
-
No canister behavior changes.
| @@ -435,6 +435,7 @@ fn handle_list_neurons( | |||
| include_neurons_readable_by_caller: true, | |||
| include_empty_neurons_readable_by_caller: None, | |||
| include_public_neurons_in_full_neurons: None, | |||
| start_from_neuron_id: None, | |||
There was a problem hiding this comment.
@dfinity/finint - I am wondering if the API needs to be updated to take advantage of the paging. It's likely going to start at a limit of 500, so inactive neurons won't be returned, which means in practice no users will use the paging. However, if a user had over 500 active neurons, they would not get them all in a single request.
There was a problem hiding this comment.
Are you saying that active neurons will always be in earlier pages than inactive neurons? How does that work with start_from_neuron_id? It can't be that inactive neurons always have larger neuron IDs, right?
There was a problem hiding this comment.
@max-dfinity IIUC, this ListNeurons request type would need to be modified to also take a start_from_neuron_id value. Again IIUC, what is returned to the Rosetta client is an OperationOutput::ListNeuronsResponse enum variant, which contains the (in this PR) updated ic_nns_governance_api::pb::v1::ListNeuronsResponse with the next_start_from_neuron_id field. It would then be up to the user calling Rosetta to pass this value in a subsequent call to the list neurons construction endpoint. @fsodre probably has a better understanding!
max-dfinity
force-pushed
the
msum/list-neurons-pagination
branch
from
6804e43 to
4fb4103
Compare
| // Populate the above two neuron collections. | ||
| for neuron_id in requested_neuron_ids { | ||
| for (count, neuron_id) in &mut requested_neuron_ids.into_iter().enumerate() { |
There was a problem hiding this comment.
This creates a somewhat surprising behavior if some of the (explicitly) requested neuron ids don't exist, since the next_start_from_neuron_id can be returned with fewer than 500 neurons.
I think it might be good to prevent >500 lookups on non-existent neuron ids, but this should probably be documented through comments.
| @@ -3350,6 +3350,12 @@ pub struct ListNeurons { | |||
| /// existing (unmigrated) callers. | |||
| #[prost(bool, optional, tag = "4")] | |||
| pub include_public_neurons_in_full_neurons: Option<bool>, | |||
| /// If this is set, it skips all neuron IDs until this value is reached, then returns | |||
There was a problem hiding this comment.
Does the neuron ID have to be an ID that exists?
Or is it allowed to pass lastNeuronIdFromPreviousPage + 1 in order to continue after the previous page?
Also it could be clearer whether the skipped neurons have higher IDs or lower IDs than the passed one. For example when fetching transactions, the newest transactions are in the first page.
There was a problem hiding this comment.
@dskloetd it doesn't have to exist, but it would need to be what's returned from the previous request. If the neuron didn't exist, it would have been in the list that was being queried.
Typically the skipped neurons will have lower IDs b/c of the BTreeMap, but that's not a guarantee of the API by design - it's only that if you call it with the same request but with 'start_from_neuron_id' that you got from the last response, it will give you another set.
There was a problem hiding this comment.
Thanks for adding this, @max-dfinity!
If a client requests multiple pages of list neurons, I suppose it is possible that the number of neurons matching the query may change in the meantime. Is this something that should be taken into account, or is the current "best effort" approach deemed good enough? I could imagine that if a new neuron is added during the requests, it would have a high ID and end up at the end of the list (in the last request)? I tried to look for more details of the approach, but neither this PR, nor the related JIRA ticket has any description, and the design linked in the epic also doesn't seem to cover this work.
| @@ -435,6 +435,7 @@ fn handle_list_neurons( | |||
| include_neurons_readable_by_caller: true, | |||
| include_empty_neurons_readable_by_caller: None, | |||
| include_public_neurons_in_full_neurons: None, | |||
| start_from_neuron_id: None, | |||
There was a problem hiding this comment.
@max-dfinity IIUC, this ListNeurons request type would need to be modified to also take a start_from_neuron_id value. Again IIUC, what is returned to the Rosetta client is an OperationOutput::ListNeuronsResponse enum variant, which contains the (in this PR) updated ic_nns_governance_api::pb::v1::ListNeuronsResponse with the next_start_from_neuron_id field. It would then be up to the user calling Rosetta to pass this value in a subsequent call to the list neurons construction endpoint. @fsodre probably has a better understanding!
No description provided.