Expose a health check endpoint

[mwasilew2]

It would be great if the collector exposed a health check that could be used for determining if the process is healthy or not.

We run this collector in ECS. For some reason, it got stuck and wasn't sending any diagnostics upstream. We had to manually restart it to make it work again. If the collector exposed a health check endpoint, we could include it in the ECS task definition which would prevent this from happening.

I imagine that a health check endpoint would be useful for other deployment methods as well.

[seanlinsley]

Hi @mwasilew2, we recently fixed an issue in #637 which might have been what you were seeing. Could you upgrade to version 0.63.0 and confirm if the collector is still getting stuck?

[msakrejda]

In addition to that, unfortunately, given the current architecture of the collector, I'm not sure if there's good places for a health check to hook into to make a meaningful assertion about the state of the process. As I commented on #644, I don't think a basic health check would have helped, and I don't see an easy path to a more advanced health check without significant re-architecting. Thoughts?

[mwasilew2]

we recently fixed an issue in #637 which might have been what you were seeing. Could you upgrade to version 0.63.0 and confirm if the collector is still getting stuck?

Thanks a lot, I'll bump our deployed version. Although, the current version was running fine for weeks, so it's possible the problem won't occur immediately (if at all).

I don't think a basic health check would have helped, and I don't see an easy path to a more advanced health check without significant re-architecting

that makes sense, I left a comment in #644 , I'm ok with closing it too