Addressing bulk ingester stuck threads #870
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
swallez
approved these changes
Aug 26, 2024
github-actions bot
pushed a commit
that referenced
this pull request
Aug 26, 2024
* signaling after successfully adding * stress test unit test * removed memory calc
github-actions bot
pushed a commit
that referenced
this pull request
Aug 26, 2024
* signaling after successfully adding * stress test unit test * removed memory calc
l-trotta
added a commit
that referenced
this pull request
Aug 26, 2024
* signaling after successfully adding * stress test unit test * removed memory calc Co-authored-by: Laura Trotta <[email protected]>
l-trotta
added a commit
that referenced
this pull request
Aug 26, 2024
* signaling after successfully adding * stress test unit test * removed memory calc Co-authored-by: Laura Trotta <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When the Bulk Ingester is configured in such a way that the number of external threads performing the
addoperation is much higher thanmaxConcurrentRequests(usually more than 50 to 1), after completing all of the request but the last one, the Bulk Ingester stops.If there's a flusher thread, it will keep adding requests one by one, and sending them one by one: this is because ~95% of the external threads will get stuck while adding, and a
sendoperation will only free one of those at a time.Replacing
signal()withsignalAll()to free the threads is a possible solution, but it's also expensive and considering this is an extreme case it would affect performances for every other case; so the chosen solution is to add anothersignal()after a successfuladdoperation, assuring that stuck threads will be released quickly if the condition that got them stuck no longer applies.Should fix #651