Practice · Tail-based sampling
What is the difference between head and tail sampling?
Sampling keeps only a fraction of traces to control cost. Head sampling decides at the start of a request, before you know if it failed, so rare errors are usually discarded. Tail sampling decides after the request finishes, so you can always keep the slow and failed requests that matter most.
How to get started with Tail-based sampling
- 1 Route spans through an OpenTelemetry Collector with the tail sampling processor.
- 2 Keep 100 percent of traces with errors and of traces slower than your latency SLO.
- 3 Keep a small random sample, such as 1 to 5 percent, of normal traffic for baselines.
- 4 Make sure all spans of a trace reach the same collector instance, for example with a load-balancing exporter keyed on trace ID.
The common mistake
What is the most common mistake with Tail-based sampling?
Using head sampling at 1 percent and expecting to find the rare failing request. It was almost certainly thrown away before it failed.
See it in action
Tracing Resolved calmly
Tail Sampling Caught the One in Two Thousand
A bug that corrupted one order in two thousand would have been invisible under head sampling. Tail-based sampling kept every failing trace, and the root cause was found in minutes.