support for alerting

[ssube]

We're using this plugin to integrate sysdig metrics with other datasources, but it doesn't seem that the plugin supports alerting. A message appears that:

The datasource does not support alerting queries

or:

tsdb.HandleRequest() error Could not find executor for data source type: sysdig

Is it correct that alerts are not supported? If so, would that be supported by/allowed on the Sysdig API side? I'm familiar with JS, but have just started looking at Grafana plugins, so I'm not sure what changes might be needed yet but would love to run some alerts from this plugin.

[davideschiera]

@ssube The sysdig data source doesn't currently support alerting, it's something we've been thinking since the first release but we haven't been able to work on yet.

What would be your use case for it? We can understand what it takes to make it working for you. And if you could help, that'd be awesome!

Thanks!

[ssube]

Our current use case is around statsd metrics. We're running the sysdig agent in a few kubernetes clusters, collecting statsd, and would like to set up alerts around those metrics. Further out, I'd also like to alert off the container-level resource and network metrics that sysdig captures.

I will start looking into what needs to be added for alerting. It looks like the API client is already implemented, so I imagine that can be used? Looking at the API docs, the data endpoint seems best.

I am also curious about sending sysdig alerts on to grafana, but our main goal is to version the dashboards and their alerts, with PRs building a new image and provisioning.

[davideschiera]

It makes a lot of sense.

Note that you could create alerts on Sysdig Monitor, and that could be more accurate than creating an alert on Grafana since the evaluation would happen over raw data in case of Sysdig, while it would happen over aggregated data in case of Grafana. In same cases it might lead to differences. Now, with this workflow you'd need to be able to see Sysdig alert events in Grafana, which unfortunately is not supported yet either. If you only need to create alerts to receive notifications (e.g. emails or Slack notifications or similar), then Sysdig Monitor could be sufficient for the use case.

That said, it's an interesting use case. I haven't worked much on it yet, but I'll be happy to help. At minimum, I believe you need to turn alerting property to true: https://github.com/draios/grafana-sysdig-datasource/blob/dev/src/plugin.json#L7. I'm not entirely sure if anything needs to be done to support alerts, the data API endpoint is already used by the plugin to fetch data, so I'd say nothing more is needed. I'm optimistic :-)

I'll give it a try in the next few days, let me know if you manage to make your progress in the meanwhile. And thanks for your help extending the datasource functionalities!

[davideschiera]

@ssube current update. By turning alerting to true in the plugin configuration you can get annotations on panels:

However, Grafana won't evaluate the alert if the panel is not currently displayed and the annotations don't get the right values.

I'll try to find out more about this, but I'm afraid we are hitting grafana/grafana#6841 which would allow the Sysdig datasource to extend the backend functionalities to expose the required hook to evaluate alerts.

[ssube]

Interesting, thanks for looking into this! I will try to set that option in one of our installs and experiment with the results. Looking into the Grafana issue you linked, it sounds like the backend support may land soon (and we've been staying up-to-date with their releases), but would need some additional work to support this.

Having the alerts fire when the panel is displayed is comical, we could put it up on a TV and make them "work" well enough (until the Grafana session expires).

[davideschiera]

Yeah, I'll need to keep an eye on the Grafana issue, it's definitely going to be useful for our datasource as well!

One thing I'll try to play around with are "annotations", and specifically having Grafana dashboards to pull Sysdig events and expose them as annotations in the panels. They would be visualized like Grafana alert events, so the output would be very similar. However, you would be able to leverage the Sysdig alerting engine which is fairly advanced.

I'd definitely recommend to take a look at Sysdig alerts to see if you can get your alerts configured (you can take a look at [the documentation](https://sysdigdocs.atlassian.net/wiki/spaces/Monitor/pages/205324292/Alerts or join the Sysdig Slack account to get live help). Keep in mind that using Grafana alerts over Sysdig data might not be as accurate as using Sysdig alerts.

I'll keep you posted on the progress to load Sysdig events in Grafana ;-)