Developing Grafana Dashboards with Jsonnet

We are currently developing KLUMPS, a set of preconfigured Kubernetes dashboards for Grafana. To define these dashboards we are using Jsonnet, a JSON templating language. This allows us to define panel templates that we can compose and parameterize, like g.queryPanel('up{job="node-exporter"}') which outputs tens of lines of JSON for a time-series chart (we’ll write another blog post about why and how we are using Jsonnet). These templates are ideal for large repetitive configurations like Grafana dashboards. This blog post describes the development setup we use to achieve a short feedback loop while working on the dashboards.

Slow Feedback on the Cluster

To monitor our own service we use Prometheus and Grafana. Both are defined as a Kubernetes deployments with their configurations inlined as a ConfigMap. This includes the dashboards, which are a big JSON blob inside our Grafana deployment YAML file. To test a dashboard modification I needed to kubectl apply that deployment configuration and wait for the configMap change to be picked up. This works really well for the occasional edit, but when writing new dashboards you want a faster feedback loop.

Reloading Dashboards

For that I’m running a local Grafana and make use of their new feature to load configurations from disk on a regular interval. The current master branch is basically a pre-5.0 version, which organizes its configuration a bit differently–calling it “provisioning”: The following dashboard load configuration needs to be placed in conf/provisioning/dashboards instead of conf/dashboards.

# dashboards.yml
- name: 'default'
  org_id: 1
  folder: ''
  type: file
  options:
    folder: /tmp/dashboards

After starting your local Grafana, it will look in the /tmp/dashboards directory for new dashboards, or reload modified ones.

Developing Dashboards with Jsonnet

Our preconfigured dashboards are organized so that the output can be used as a Kubernetes ConfigMap to be used by Grafana, which means the output contains all dashboards. To work on a single dashboard, we need to process the output. Say we want to work on the k8s-cluster-rsrc-use dashboard. The dashboards can be compiled using the jsonnet command:

jsonnet klumps.libsonnet

The output looks something like this:

{
   "k8s-cluster-rsrc-use.json": {
      "annotations": {
         "list": [ ]
      },
      "editable": true,
      "gnetId": null,
      "graphTooltip": 0,
      "hideControls": false,
      "id": 1,
      ...

There are two problems with this output: 1. Grafana does not understand groups of dashboards in one file (we only want the dashboard k8s-cluster-rsrc-use anyway), and 2. the dashboard has an id set which interferes with Grafana’s reloading logic.

Luckily we can post-process the JSON with jq, to select only the dashboard we want, and to delete the id field:

jsonnet klumps.libsonnet | jq '.["k8s-cluster-rsrc-use.json"] | del(.id)' \
    > /tmp/dashboards/klumps.json

The result is a single dashboard JSON file placed where Grafana has been configured to look for dashboards. Roughly five seconds later Grafana picks up the change and you can find the new dashboard in the dashboard list, or you can reload the tab if you are looking at the dashboard already (note that Grafana uses the dashboard title as the URL for the dashboard, if you change the title, you need to adapt the URL).