Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add timeout to osqueryinstance healthchecks #2050

Open
zackattack01 opened this issue Jan 16, 2025 · 0 comments · May be fixed by #2051
Open

Add timeout to osqueryinstance healthchecks #2050

zackattack01 opened this issue Jan 16, 2025 · 0 comments · May be fixed by #2051

Comments

@zackattack01
Copy link
Contributor

Some logs today showed the following on a windows device running launcher v1.12.3:

  • all osqueryinstance errgroup routines starting correctly
  • eventually, osquery stops logging and localserver started logging the following errors:
    • "msg":"updating id fields","component":"localserver","err":"id query failed: could not query the extension manager client: i/o timeout"
  • this state continued for days, without the healthcheck routine triggering an osquery restart (or logging any sort of failure)
  • when a shutdown was finally issued, the healthcheck goroutine never returned

I think the most likely explanation here is that it got caught up somehow, and that we should enforce some sort of cancellation on our Healthy() calls. This way we will at least log the failures and restart to eventually recover from this state

@zackattack01 zackattack01 linked a pull request Jan 17, 2025 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant