OpenCensus with Prometheus and Kubernetes

Yesterday, Google announced OpenCensus, an instrumentation framework for monitoring and tracing. It comes with a set of client libraries for Golang and Java, with more to come. More importantly, it introduces a set of abstractions (records, views, exporters) that could shape how apps will be instrumented.

Today we’ve been kicking the tyres by building their example Go app and connecting it to a local Prometheus.

Get the Example App Running

OpenCensus’ Golang instrumentation comes with an example app that shows how to take measurements, and then expose them to a /metrics endpoint for Prometheus:

# Clone the repo
go get github.com/census-instrumentation/opencensus-go
# Go to the directory
cd $GOPATH/src/github.com/census-instrumentation/opencensus-go/examples/stats/prometheus/
# Install dependencies
go install ./...
# Build the example app
go build -o app .
# Run the app
./app

The app now runs and takes measurements in an infinite loop. It exposes the /metrics endpoint on port 9999, we can verify this in the browser: http://localhost:9999/metrics.

Looking at the Instrumentation

At this point it’s worth looking at the code of main.go to see what’s going on:

Exporters make the metrics consumable by another service. This exporter is also an http handler that will expose the /metrics endpoint.

exporter, err := prometheus.NewExporter(prometheus.Options{})
stats.RegisterExporter(exporter)
http.Handle("/metrics", exporter)

Note that the concept of an “exporter” here is different to exporters in Prometheus land, where it usually means a separate process that translates between an applications native metrics interface and the prometheus exposition format, running as a sidecar or colocated in some other way with the application.

Before we can measure anything, the measure type needs to be registered. Note that a measure does not necessarily end up in an exporter. Therefor the measure’s name and description will not end up in Prometheus’ exposition format. To record a single measurement we need to obtain a measure reference either by creating a new one, or by finding one via FindMeasure(name).

videoCount, err := stats.NewMeasureInt64("my.org/measures/video_count", "number of processed videos", "")
stats.Record(ctx, videoCount.M(1))

This increased the videoCount counter by 1. If you read the documentation carefully you’ll notice that this recording would be ignored because we have not registered the measure as part of a view:

viewCount, err := stats.NewView(
    "video_count",
    "number of videos processed processed over time",
    nil,
    videoCount,
    stats.CountAggregation{},
    stats.Cumulative{},
)
viewCount.Subscribe()

After a view like the above uses the videoCount measure, its recordings will be kept, and by calling Subscribe() it makes its metrics available to the exporters. In NewView() we also define the metric name and the description that will end up as HELP in Prometheus’ exposition format. The metric name will be prefixed by the exporter’s Options.Namespace, which by default is opencensus. To overwrite it, the exporter needs the namespace set:

prometheus.NewExporter(prometheus.Options{Namespace: "myapp"})

Adding Labels

New labels can be added via tags. Say we want to add a route label, for this we need to declare it and register it in the context:

routeKey, err := tag.NewKey("route")
if err != nil {
    log.Fatal(err)
}

tagMap, err2 := tag.NewMap(ctx,
    tag.Insert(routeKey, "/myroute"),
)
if err2 != nil {
    log.Fatal(err2)
}
ctx = tag.NewContext(ctx, tagMap)

This augmented context is later used in the stats.Record() call where it passes on those tags as labels. Then we need to tell the view which tags we want as labels, here we only pass one: routeKey:

viewCount, err := stats.NewView(
    "video_count_total",
    "number of videos processed processed over time",
    []tag.Key{routeKey},
    videoCount,
    stats.CountAggregation{},
    stats.Cumulative{},
)

Finally we can see a line like the following under http://localhost:9999/metrics:

myapp_video_count_total{route="/myroute"} 19

To see the metrics in your local Prometheus, add the app as a scrape target to your Prometheus configuration YAML:

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'example-app'
    static_configs:
      - targets: ['localhost:9999']

Now you can see them in Prometheus’ expression browser: http://localhost:9090/

OpenCensus and Kubernetes

Once the app is containerized and deployed into Kubernetes, it keeps measuring while it’s running. The Prometheus that runs in your Kubernetes cluster can scrape it like any other target by virtue of its /metrics endpoint. If your Prometheus configuration has the relevant relabeling rules defined, the metrics will get the appropiate pod and job labels, see Prometheus and Kubernetes: Monitoring Your Applications on how this can be achieved.

Needless to say that OpenCensus-instrumented apps will work well with Kausal as they can be scraped by Prometheus.

It will be interesting to see if libraries adopt the OpenCensus measures, which always record, but which have to be added to views in the app in order for them not to be dropped.