kube-prometheus ships with a set of default alerting rules and dashboards. At some point one might like to extend them. This document is intended to explain the workflow of how additional alerting rules and dashboards could be added.
For both, the Prometheus alerting rules as well as the Grafana dashboards, there are Kubernetes ConfigMaps, that are generated from content in the assets/ directory.
The source of truth for the alerts and dashboards are the files in the assets/ directory. The respective files have to be changed there and then the make generate make target is executed to re-generate the Kubernetes manifests.
Note: make generate should be executed from kube-prometheus base directory.
The ConfigMap that is generated and holds the alerting rule files can be found in manifests/prometheus/prometheus-k8s-rules.yaml.
It is generated by taking all the *.rules files in the assets/prometheus/rules/ directory and generate the ConfigMap.
To extend the alerting rules simply add a new .rules file into the assets/prometheus/rules/ directory and re-generate the manifests. To modify the existing rules, simply edit the respective .rules file and re-generate the manifest.
Then the generated manifest can be applied against a Kubernetes cluster.
The ConfigMap that is generated and holds the dashboard definitions can be found in manifests/grafana/grafana-dashboards.yaml.
As Grafana's support for applying dashboards from files is limited a sidecar (called "grafana-watcher") was implemented. It watches the dashboard definitions provided through the ConfigMap and ensures that Grafana's SQLite database is in sync with the dashboard definitions.
To edit/create a dashboard login to Grafana and modify and save the dashboard. Then download the dashboard definition in Grafana through Share -> Export -> Save to file. Move the file to assets/grafana/ and re-generate the manifests (executing make generate from kube-prometheus base directory).
Note: The dashboard json file to be copied in assets/grafana/ should be suffixed with -dashboard.json, otherwise it won't be processed by make generate.
Then the generated manifest can be applied against a Kubernetes cluster with something like:
kubectl -n monitoring apply -f manifests/grafana/grafana-dashboards.yaml
That will update the ConfigMap grafana-dashboards. Change should be automatically detected by grafana-watcher and dashboards reloaded.