OpenTelemetry Collects the Data. Who Makes Sense of It?

OpenTelemetry solved a problem the industry had for a decade: every vendor had its own agent, its own SDK, and its own wire format, and switching tools meant re-instrumenting everything. Today you can instrument once, export over OTLP, and send the data to almost any backend, open source or commercial.

That is a big deal. It is also easy to mistake for the whole job.

The Signal

Picture a team six months into an OpenTelemetry rollout. Every service has the SDK. The Collector runs in every cluster. Traces, metrics, and logs are flowing. The migration ticket is closed.

Then an incident hits. Latency on the orders API doubles. The on-call engineer has more telemetry than ever, and it still takes forty minutes to find the cause, because:

• the metrics call the service orders, the traces call it orders-api, and the logs call it orders-7f9c5d8b4-x2kqp;
• the logs have no trace IDs, so there is no way to jump from a slow trace to what the service printed during it;
• a 1% head sampling rate means the slow requests were mostly dropped before anyone knew they would be slow.

All of the data was collected. Very little of it could be connected.

The Problem

OpenTelemetry standardizes how telemetry is produced and transported. It does not decide, on your behalf:

• which attributes every signal must carry so they can be joined;
• how logs get trace context;
• what to sample, and where that decision is made;
• where data is stored, for how long, and at what cost;
• how an engineer moves from a symptom to a cause during an incident.

Those are architecture decisions. If nobody makes them on purpose, they get made by default, one service at a time.

Beyond the Dashboard

A dashboard shows you one signal at a time. An investigation moves between signals: from a latency spike (metric) to an example of a slow request (trace) to what the code said during it (log), and sometimes to where it spent CPU or memory (profile).

Every one of those jumps needs a shared key:

Jump	What makes it possible
Metric to trace	Exemplars on histograms, or the same service.name and route on both
Trace to logs	trace_id and span_id on every log record written during a span
Any signal to infrastructure	The same resource attributes, such as k8s.pod.name and k8s.namespace.name
Service to service	Context propagation (W3C traceparent) across every hop, including queues

If any of these keys is missing, the jump becomes a manual search by timestamp.

The Investigation

When an OpenTelemetry deployment produces plenty of data but slow investigations, these checks usually find the gaps:

1. Resource attributes. Pick three random services and compare service.name, service.version, and deployment.environment.name across their traces, metrics, and logs. They should match exactly.
2. Log correlation. Search your logs for records with a trace_id. If the share is low, the logging library is not bridged to the OpenTelemetry context, or logs are scraped from files without it.
3. Propagation. Pick a request that crosses a queue or an async job. Does the trace continue on the consumer side, or does it start a new, orphaned trace?
4. Sampling. Find the slowest requests from yesterday in your load balancer logs, then look for their traces. If most are missing, sampling is deciding too early.
5. Query. Time how long it takes to answer “show me the logs for the five slowest checkout traces in the last hour.” That is the number to improve.

The Engineering Decision

A common, vendor-neutral architecture uses two tiers of Collectors:

• Agents (in Kubernetes, typically a DaemonSet) receive OTLP from local pods, add Kubernetes metadata, and forward data.
• Gateways (a Deployment) do the work that needs to see whole traces, such as tail sampling, plus routing and export to one or more backends.

A trimmed agent configuration might look like this:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  k8sattributes:
    # In agent mode, only watch pods on this node (K8S_NODE_NAME comes from
    # the downward API in the DaemonSet spec).
    filter:
      node_from_env_var: K8S_NODE_NAME
    extract:
      metadata:
        - k8s.namespace.name
        - k8s.pod.name
        - k8s.deployment.name
        - k8s.node.name
  batch: {}

exporters:
  # Traces: route by trace ID, so every span of a trace reaches the same
  # gateway replica and tail sampling sees whole traces.
  loadbalancing:
    routing_key: traceID
    protocol:
      otlp:
        tls:
          insecure: true # in-cluster example only; use TLS across trust boundaries
    resolver:
      k8s:
        service: otel-gateway.observability
  # Metrics and logs: any gateway replica will do.
  otlp:
    endpoint: otel-gateway.observability.svc:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [loadbalancing]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [otlp]

A few things this sketch leaves out. The k8sattributes processor and the loadbalancing exporter’s Kubernetes resolver both need RBAC permissions: pods, namespaces, and replica sets for the processor (to resolve deployment names), and endpoints or EndpointSlices for the resolver, depending on its version. The gateway, not shown, runs the tail_sampling processor; that only works because the agent routes every span of a trace to the same gateway replica. And if you add the resourcedetection processor on an agent, check its override setting, so it does not replace the application’s own host.name with the Collector pod’s. Check the component documentation for the versions you run; options change between releases.

Beyond the pipeline, the decisions that matter most are:

Agree on a resource attribute contract. Write down the attributes every signal must carry, enforce them in the Collector, and treat a mismatch as a bug.

Put trace context in logs. Use your language’s OpenTelemetry log bridge, or at least inject trace_id and span_id into structured logs.

Sample late, keep what matters. Keep all errors and slow traces with tail sampling, and sample the healthy majority. The tail sampling story shows why.

Choose a backend by how it supports investigation. Many backends accept OTLP today, from open source projects such as Jaeger, Prometheus, the Grafana stack, and OpenObserve to commercial services such as Datadog, Honeycomb, New Relic, and Elastic. OTLP support is table stakes. The questions that separate them for your team are:

Question	Why it matters
Can I go from a metric to traces to logs without copying IDs by hand?	That jump is most of an investigation
What does high cardinality cost me?	Unbounded attributes are where telemetry bills and query times go wrong
How long can I keep each signal, at what price?	Retention decides which questions you can answer next month
What query language does it use, and can my team write it under pressure?	The best data is useless if nobody can query it at 3am
Can I run it where my data must live?	Data residency and security rules may decide for you

There is no universally right answer. A small team may value one place for all signals over best-of-breed components. A large platform team may prefer to compose specialized backends. OpenTelemetry’s real gift is that you can change this decision later without re-instrumenting.

The Outcome

This is a hypothetical example, so there is no measured outcome to report. To check whether your own changes are working:

• Re-run the investigation checks above every quarter and track the results.
• Measure the share of log records that carry a trace_id.
• In incident reviews, note how long it took to go from the first alert to the first useful trace, and watch that number over time.

Lessons from the Field

• OpenTelemetry standardizes collection and transport. Correlation is still a design decision.
• A shared resource attribute contract is the cheapest, highest-leverage observability investment most teams can make.
• Logs without trace context are a separate tool, however close they sit to your traces.
• Decide where sampling happens before an incident decides for you.
• Pick backends by how quickly they get an engineer from symptom to cause, not by how many signals they can ingest.