Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Get processing timeline events #3241

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from
Draft

Get processing timeline events #3241

wants to merge 3 commits into from

Conversation

jbaptperez
Copy link

@jbaptperez jbaptperez commented Dec 5, 2024 •
edited
Loading

IMPORTANT: All Pull Requests should be connected to an issue, if you don't
have an issue, please start by creating an issue and link it to the PR.

Please provide enough information so that others can review your pull request:

  • What existing problem does this PR solve?
    • Impossible to get timeline events when the one is being updated (new Plaso ingestion for the timeline).
  • What new feature is being introduced with this PR?
    • Allows to get timeline events when the timeline is being updated (new Plaso ingestion) i.e. allows to get partial events, already indexed,
    • Adds a setting to enable or disable this feature,
    • The UI clearly communicates to the user that search results against processing timelines may be incomplete and subject to change.
  • Overview of changes to existing functions if required.
    • Backend: In the Sketch model, the active_timelines property includes timelines with a status set to processing,
    • Backend: In the utils.py, the get_validated_indices function includes timelines with a status set to processing,
    • Backend: The POST /sketches/{{sketch_id}}/aggregation/explore/ endpoint includes indices which timeline has a status set to processing.

Checks

  • All tests succeed.
  • Unit tests added.
  • e2e tests added.
  • Documentation updated.

Closing issues

Closes #3219.

@jbaptperez jbaptperez force-pushed the get-processing-timeline-events branch 2 times, most recently from 42403d9 to de22d67 Compare December 5, 2024 17:09
@jkppr jkppr added the Backend label Dec 12, 2024
@jkppr
Copy link
Collaborator

jkppr commented Dec 12, 2024

Thanks for adding a draft PR for early feedback. If I see this correctly, you are also having the changes submitted in your PR #3226 in this PR. Can you please separate this out and only commit the changes relevant to the issue #3219 in this PR?

This would make reviewing the code much easier, thanks.

@jbaptperez jbaptperez force-pushed the get-processing-timeline-events branch from de22d67 to 281bb34 Compare December 14, 2024 15:09
@jbaptperez
Copy link
Author

@jkppr, I removed the work of the other PR.
I also rebased the branch onto master.

The only change I made for now is at the backend side: It unlocks the desired behaviour: Getting timeline events while in the processing status.
Note for now I did not added any frontend or backend setting to properly toggle this feature, I'll work on it at the end.

At the frontend side, for now, all timeline events appear even if some of them are is in the processing state.
It is easy to reproduce such a situation by forcing a timeline state in the database.
This results in an infinite spinning wheel in the timeline chip.

However, I face difficulties when attempting to change the timeline chip so that a processing one behaves like a ready/failed one, with an additional spinning wheel as the only visual difference.

The complementarity between TimelineChip.vue and TimelineComponent.vue is not trivial.

Do you think someone can help me to accomplish this precise part?

@Annoraaq
Copy link
Collaborator

TimelineComponent.vue contains a slot that is shown if the timeline status is processing. My understanding is that you want to change that and instead want to render the other slot (called "processed" in the code).

Now one thing to be careful about is that TimelineComponent.vue is used by Explore/TimelineChip.vue and Analyzer/TimelineChip.vue and whatever you change in TimelineComponen.vue will impact both types of timeline chips.

@jbaptperez jbaptperez force-pushed the get-processing-timeline-events branch 2 times, most recently from bfcfffd to 7aad31c Compare December 23, 2024 14:04
@jbaptperez jbaptperez force-pushed the get-processing-timeline-events branch from 7aad31c to 0177d57 Compare December 23, 2024 17:25
@jbaptperez
Copy link
Author

jbaptperez commented Dec 23, 2024 •
edited
Loading

Hi @jkppr, I have made progress, but still WIP.

For now:

  • I added a user setting (false by default): showProcessingTimelineEvents,
  • I adapted the frontend-ng so that it properly reads the setting,
  • The frontend can display processing timeline events with a consistent TimelineComponent/TimelineChip set,
  • I have manually checked that there is no issue when displaying the timeline chip in the_Analyser results_ and the Visualizations pages.

I am currently developing the following:

  • Updating the backend so that the current hard-coded changes in my branch become variable, depending on a default value as a method parameter or coming from the related endpoint forms (at least GET /sketches/{sketch_id}/ and POST /sketches/{sketch_id}/explore/),
  • Updating the frontend (store.js, RestApiClient.js) so that the setting is properly dispatched over the views.

Feel free to give me advice or instructions to fine tune my change, if necessary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@jbaptperez jbaptperez

@jkppr jkppr

Labels
Backend
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Seeing data already ingested into a timeline when the related search index is being updated
3 participants
@jbaptperez @jkppr @Annoraaq