Add page information for datasets to the json

[SarahSidders]

Hello, we're currently building a UI for our version of ERDDAP, and we want to be able to bring back the total number of datasets a query will return. The issue we are currently experiencing is that information such as the total dataset count can only be retrieved in the html. Is it possible for the current page, total number of rows, number of pages, and total count, to also be included in the json?

Example html with total dataset count:
https://coastwatch.pfeg.noaa.gov/erddap/search/advanced.html?page=1&itemsPerPage=1&searchFor=argo

Example json without total dataset count:
https://coastwatch.pfeg.noaa.gov/erddap/search/advanced.json?page=1&itemsPerPage=1&searchFor=argo

By having this information in a machine-readable form, then we will be able to retrieve this data in a more consistent way.

Many thanks,

Sarah

[ChrisJohnNOAA]

I think adding that information to the response is reasonable and I welcome you to make a contribution to do that. That said since the output is currently only writing a table to JSON and nothing else, adding additional information to the response is more complicated than you might expect. If you would like to pursue this it would be good for you to share how you plan to implement the changes before getting started.

The handling for advanced search is here: https://github.com/ERDDAP/erddap/blob/main/WEB-INF/classes/gov/noaa/pfel/erddap/Erddap.java#L11506

[BobSimons]

I'm not sure of your goals, but would it work to simply set itemsPerPage=1000000 (more than are in the ERDDAP) in order to simply get all of the results at once? That is the most efficient way for ERDDAP to provide all of the results to you (instead of ERDDAP running the query repeatedly, once per page request, and each time returning part of the results). Then you can simply count the number of matching datasets. The itemsPerPage system was devised to avoid trouble with browsers when they get pages that are too big, and when human users think they are most likely to find the dataset they want on the first page (e.g., one dataset out of the first 1000).

I'll add: if you make multiple queries (e.g., with itemsPerPage=1 or 100), the number of matching datasets may change slightly as datasets temporarily drop out or are reloaded after having dropped out. A single query getting the entire response avoids this problem.

I'll add: that said, when the ERDDAP has a huge number of datasets, you may want to put some sort of limit on the max number of datasets in the response (e.g., itemsPerPage=10000) and just use the results on that first page. If a request got 10000 matches, is someone really going to look at all of the results? Or is it a sign that they need to refine their search?