Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gaps in samza data #14

Open
ngpadp12 opened this issue Feb 10, 2022 · 2 comments
Open

Gaps in samza data #14

ngpadp12 opened this issue Feb 10, 2022 · 2 comments

Comments

@ngpadp12
Copy link

Some data points are not displaying a continuous graph.

Raising this issue to understand why it might be happening.
Please go through the following screenshots:

Round robin chooser:
image

YARN
image

@nmaquet
Copy link
Contributor

nmaquet commented Feb 10, 2022

@ngpadp12 I can try and help you troubleshoot this but I would need a bit more information. What do the exporter logs say? Is the exporter crashing and restarting? Is Prometheus properly scraping the metrics regularly? Could you maybe share the graph of scrape_duration_seconds in Prometheus to see what's happening?

@ngpadp12
Copy link
Author

ngpadp12 commented Feb 14, 2022

Hello @nmaquet , thanks for responding!

The only logs I see are of unrecognized metrics. But I am not sure if the unrecognized metrics can ever show up for prometheus to scrape(and hence the gaps being the times when the same metrics goes unrecognized)
Screenshot below:
image

We are running this as an AWS ECS service, and no, the exporter is not crashing and restarting.
Image below shows the time since when it is up and running:
image

The scrape duration graph of the past week is below:
image

Is Prometheus properly scraping the metrics regularly?
I believe yes, it is. But is there any parameter or anything in particular that you want me to check to confirm this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants