Exemplars in practice: Finding the needle in your observability haystack

A common desire among SREs is to have the ability to quickly jump between different types of telemetry data, for example being able to jump from a dashboard showing a metric to the distributed traces that the metric represents. Observability vendors have come up with various proprietary approaches to meet this need with varying degrees of success, but open-source solutions have started to gravitate towards the idea of exemplars to provide a common solution to this problem.

In this talk, Joel will give a brief introduction of exemplars and to the open-source tracing landscape (e.g. OpenTelemetry), and cover how exemplars give engineers a straightforward path to jump from a metric on a dashboard to a trace related to that metric - ultimately allowing them to be more efficient. He will also explore some limitations of using exemplars, namely:

1. Exemplars provide the means to navigate directly to a single trace, but not a population of traces. This limits their usefulness to solving problems where examining a single request gives enough context to a user, which is rarely true.

2. Exemplars push the burden of identifying the “relevant” traces to publish onto engineers who have to determine how to sample traces at runtime in applications to get the best results. This becomes a frustrating exercise when coupled with the fact that most problems requiring distributed traces are multifaceted in nature. In other words, it is impossible to pick a single trace that has all of the relevant details that a user or SRE wants to see.

Joel will also present an alternative method to exemplars for linking metrics and distributed traces that addresses the problems that have been discussed by giving engineers the ability to create the link between a dashboard and a query to analyze all of the relevant traces they have collected from their applications.