Prevent opensearch from aggregating across all indices. #3192
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses an issue where Timesketch attempts to execute an OpenSearch query with an empty index list when fetching labels for sketches without active timelines. This defaults to querying
_all
indices, which can lead to performance issues and the "got more than 100 headers" error due to excessive task management headers and/or large responses in the aggregation.The following changes were made:
SketchResource.get
function (timesketch/api/v1/resources/sketch.py
), a conditional check was added to ensureget_filter_labels
is called only ifsketch_indices
is not empty. If the list is empty (no indices to query), an empty list is directly assigned tofilter_labels
in the metadata.get_filter_labels
function (timesketch/lib/datastores/opensearch.py
) a check was added at the beginning of the function to immediately return an empty list if the providedindices
parameter is empty. This prevents the aggregation query from being constructed and executed unnecessarily. Explanatory comments were added to clarify the purpose of these changes.These modifications prevent querying
_all
indices with emptysketch_indices
, improving performance and preventing header limit errors when fetching sketch details for empty sketches or sketches without active timelines.Closing issues
closes #3191