Improve performance of TimelineFieldsResource endpoint

[jkppr]

In PR #3182 we introduced a new API endpoint TimelineFieldsResource to gather unique field names per timeline to improve the UX of the visualizations dialog.

https://github.com/google/timesketch/pull/3182/files#diff-bb4334f2502d8d96c5689e099dd1440470e425b43a01d8221ebfaabd18511862R538

Currently, the endpoint retrieves the unique fields by:

1. Aggregating all data types within the timeline.
2. For each data type, querying for a single event to get the fields.

This approach can be inefficient, especially for timelines with many data types. This approach also ignores that some data_types can have different fields per event or have some events with annotated fields by analyzers.

This issue is to track research into a more efficient way of gathering those fields. Possible approaches to investigate include:

• Using OpenSearch field capabilities: Explore if OpenSearch provides a more direct way to retrieve unique fields for a given index and data type.
• Caching field information: Investigate caching the field information to avoid redundant queries.
• Optimizing the aggregation: Analyze if the aggregation step can be optimized or if it can be avoided altogether.
  • Find all unique fields for a given data_type

Tasks:

• Research alternative approaches for retrieving unique fields.
• Benchmark the performance of different approaches.
• Implement the most performant solution.

Related PRs:

• Add query string filtering to Visualizations #3182

[manojks1999]

@jkppr Can I work on this ?

[jkppr]

Hi @manojks1999 , before working on this problem, let's brainstorm your ideas on how to tackle the problems outlined. Can you share your ideas/concepts on how to fix these problems?