Document how to programmatically access Grafana / Prometheus data

[choldgraf]

Context

We currently collect data with Prometheus to track various usage metrics across our hubs, and are lengthening that to a year. This information is accessible in Grafana dashboards as documented here.

However, I don't believe we have it documented how a person could access and use this data any other way than via the Grafana UI.

Proposal

We should document how a person with little prior experience could perform a few critical tasks with our Grafana / Prometheus data on any given cluster:

• Access the raw data
• Run basic queries for common usage metrics (and link out to some PromQL guides if necessary?)
• Split up Grafana usage by hub and component
• Perform some basic visualizations

Updates and actions

https://team-compass.2i2c.org/partnerships/community_success/hub-activity/

[yuvipanda]

@choldgraf we do have documentation here: https://infrastructure.2i2c.org/en/latest/howto/operate/grafana.html?highlight=grafana

Prometheus only collects and stores metrics, it is accessed via Grafana. Prometheus itself doesn't have any significant UI. Grafana dashboards are made of promql queries that talk to our prometheus instance to get the data.

I'm not sure if this is an issue about documenting our specific Grafana dashboards some more (for which we have #1117), or something else.

[damianavila]

I'm not sure if this is an issue about documenting our specific Grafana dashboards some more (for which we have #1117), or something else.

I think that is the case, but let's give @choldgraf a few days to confirm it and eventually close this one in favor of #1117.

[choldgraf]

Ah ok - I've re-named / scoped this issue then. My goal here is to be able to programmatically access the Grafana data, or download raw data to use in visualizations, notebooks, etc. I can't find that information anywhere in our docs.

[yuvipanda]

I've edited the title, and here's some code that will:

1. Let you give it the name of a cluster (like utoronto)
2. Will figure out authentication to help you run arbitrary promQL queries against that cluster
3. Return pandas dataframes so you can do pandas things to it.

In this example notebook, I'm basically recreating this graph with pandas https://grafana.utoronto.2i2c.cloud/d/Qx58n92nk/cluster-information?orgId=1&viewPanel=3&editPanel=3

https://notebooksharing.space/view/42a145c3c519dcf27b477e09744a79baef1f5831b37f543cdb02e46ed9d95c8b#displayOptions=

@choldgraf is this the kinda thing you wanted? ^

As just a point of nomenclature (and something to be clarified via #1891), grafana itself contains no data at all in our setup - in other folks setup it is possible that grafana itself has data. In our case, data is only in prometheus and not in grafana at all, so just wanted to avoid that confusion.

[choldgraf]

yes this is just the kind of thing that I had in mind! And sorry for the naming confusion - I had called it Prometheus in the first version of this issue but people told me that Grafana was where we visualized data instead. I think that this was actually just a mis-understanding about the nature of my request. What you've done is much closer to what I had in mind!

cc @jmunroe who might be interested in using workflows like what @yuvipanda shows above to hack on some dataviz :-)

[yuvipanda]

I documented this in https://hackmd.io/NllqOUfaTLCXcDQPipr4rg

[choldgraf]

Thanks so much for providing a link to that HackMD - I've cross-linked it from another issue where I think this would be most immediately useful just as an FYI:

• Display historical hub usage data as a heatmap over time by using Prometheus data kpis#27

That also gives more explanation for why this kind of data could be useful (for generating little MyST reports)

[jnywong]

The HackMD guide has been transformed into this basic guide in the Team Compass:

https://team-compass.2i2c.org/partnerships/community_success/hub-activity/

@choldgraf has also updated https://2i2c.org/kpis/cloud/#active-users using this guide.

@choldgraf I hope this issue can be considered resolved, but if there are any other dangling bits let me know 😄

[choldgraf]

@jnywong it's your call if you think this issue is resolved now - in general I think it's safe to just close issues + provide a rationale (like the one above) for how it is resolved.

[jnywong]

Huzzah!

[yuvipanda]

Amazing work, @jnywong! Thank you!!!