Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ai/live: Error handling and retries for event subscriptions. #3338

Merged
merged 2 commits into from
Jan 7, 2025

Conversation

j0sh
Copy link
Collaborator

@j0sh j0sh commented Jan 3, 2025

Added this after noticing a number of errors in the prod logs for the event subscriber along the lines of

Channel exists but sequence does not: requested 71 latest 71

This is harmless and usually happens at the very end of the stream when a publisher cancels a segment just before closing the entire channel, but I thought it would be best to make the overall behavior a little more robust.

Specifically:

  • Exits immediately on EOS / StreamNotFound
  • Retries 5 times max
  • Retry counter resets to zero after each successful read
  • 300ms sleep in between each retry
  • Stops the pipeline if max retries hit

@github-actions github-actions bot added go Pull requests that update Go code AI Issues and PR related to the AI-video branch. labels Jan 3, 2025
Copy link

codecov bot commented Jan 3, 2025

Codecov Report

Attention: Patch coverage is 0% with 29 lines in your changes missing coverage. Please review.

Project coverage is 33.72469%. Comparing base (cfc7b6e) to head (4fa7bfd).
Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
server/ai_live_video.go 0.00000% 26 Missing ⚠️
trickle/trickle_server.go 0.00000% 3 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@                 Coverage Diff                 @@
##              master       #3338         +/-   ##
===================================================
- Coverage   33.74545%   33.72469%   -0.02076%     
===================================================
  Files            141         141                 
  Lines          37368       37391         +23     
===================================================
  Hits           12610       12610                 
- Misses         24038       24061         +23     
  Partials         720         720                 
Files with missing lines Coverage Δ
trickle/trickle_server.go 0.00000% <0.00000%> (ø)
server/ai_live_video.go 0.00000% <0.00000%> (ø)

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cfc7b6e...4fa7bfd. Read the comment docs.

Files with missing lines Coverage Δ
trickle/trickle_server.go 0.00000% <0.00000%> (ø)
server/ai_live_video.go 0.00000% <0.00000%> (ø)

@j0sh j0sh merged commit 17832e7 into master Jan 7, 2025
18 checks passed
@j0sh j0sh deleted the ja/event-sub-retries branch January 7, 2025 18:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
AI Issues and PR related to the AI-video branch. go Pull requests that update Go code
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants