-
Notifications
You must be signed in to change notification settings - Fork 35
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: re-sync on drifted head + sentry alert #1332
Conversation
While doing a small performance check I was able to encounter a very similar issue to the one reported in #1333, I was running the node and it suddenly stoped for some minutes. Just restarting it solved the issue, because the Sync kicked in, following that logic i added a quick work-arround to re-sync when the drift happens, this proved to work as expected in a test and might be able to avoid #1333 in the future. Unfortunately we'll need to monitor this changes live and make sure the issue actually stopped. Here is a small log output showing how it worked in a simple scenario of a small delay.
|
block_procesing.exs
fixThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Motivation
We had an issue with the node stopping without addition information nor any errors on sight, we want to add an alert related to it that can be picked up by Sentry.
Description
This PR adds an Alert for when state transitions stop and the head start to drift. This is accomplished using the already present
:syncing
keyword in theLibP2PPort
state and just error when we detect the first time the head drifted by a predefined amount, now set to 12.It also deals with a small change needed to run the
block_processing.exs
file after the addition of theStoreStatesSupervisor
.After reproducing #1333 in a Sepolia run in my local machine I was able to resume execution when re-dyncing, now it might solved.
Resolves #1333 (We need to validate this live and reopen the issue if it happens again)