Your Dashboard Is Green. Your Users Are Still Suffering.

Every engineer has seen this screen. CPU is fine, memory is fine, every pod is Running, the node pool is healthy, and the service dashboard is a calm wall of green. Meanwhile, the support queue is filling with people who cannot check out.

Nothing on that dashboard is wrong. It is answering a different question from the one users are asking.

The Signal

In our example, the first signal does not come from monitoring at all. It comes from a support ticket: “Checkout spins for a long time and then fails. Trying again sometimes works.”

The on-call engineer opens the checkout dashboard and sees:

Panel	Value	Status
CPU utilization (all pods)	41%	Green
Memory utilization	58%	Green
Pods ready	12 / 12	Green
Average request latency	180 ms	Green
HTTP 5xx rate	0.2%	Green

Every number is true. Users are still having a bad time.

The Problem

A green infrastructure dashboard tells you the machinery is running. It says very little about whether the work users care about is succeeding. In our example, four ordinary effects add up:

1. Averages hide tails. An average of 180 ms is compatible with 1% of requests taking ten seconds, if the rest take about 80 ms. Those requests belong to real people, and one checkout in a hundred taking ten seconds is a lot of abandoned carts.
2. Errors can be counted in the wrong place. The application records its own status codes. Requests that time out at the load balancer, are reset in the connection pool, or never reach a pod do not appear in the application’s 5xx rate at all.
3. Retries turn failures into latency. A client that retries a failed call once may succeed on the second try. The server sees one error and one success, and the user sees a six-second spinner.
4. Health checks test the process, not the path. A /healthz endpoint that returns 200 OK proves the process can answer HTTP. It does not prove the payment provider, the inventory service, or the database connection pool are working for real requests.

Beyond the Dashboard

The dashboard was built from the inside out. It measures resources (CPU, memory, pod counts) and the service’s own view of its requests. Each panel is a cause-side signal: useful for explaining a problem once you know you have one, poor at telling you that users are hurting.

User experience is measured from the outside in. The question is not “is the service up?” but “did the user’s checkout succeed, and how long did it take?” To answer that, you need service level indicators (SLIs) measured as close to the user as you can get:

• At the edge: load balancer or ingress logs and metrics, which see timeouts and resets that the application never sees.
• In the client: real user monitoring (RUM) or mobile telemetry, which sees DNS, TLS, CDN, and front-end errors.
• Synthetically: a scripted checkout that runs every minute from outside your network, which tells you the journey works even when traffic is low.

The Investigation

With the question reframed, the investigation looks different.

Start at the edge. Compare the request count and error rate seen by the load balancer with what the application reports. A gap between the two is itself a finding: requests are failing before your code runs.

Look at the distribution, not the average. With OpenTelemetry HTTP instrumentation, request durations are recorded as a histogram. In Prometheus naming, the semantic-convention metric http.server.request.duration usually appears as http_server_request_duration_seconds. The 99th percentile per route is one query away:

histogram_quantile(
  0.99,
  sum by (le, http_route) (
    rate(http_server_request_duration_seconds_bucket{job="checkout"}[5m])
  )
)

Exact metric and label names depend on your instrumentation and exporter versions. Prometheus usually maps service.name to the job label, so check what your pipeline actually emits.

Break it down by dimension. A problem that affects 3% of requests overall might affect 60% of requests for one region, one payment method, or one tenant. Group by the attributes you have: route, region, availability zone, version, customer tier.

Follow an example. Once you know where the slow requests are, pull a handful of traces from that slice. Exemplars, or a trace search filtered on duration and route, take you from “p99 is bad” to “this call to the payment provider waits on a connection pool for four seconds, then retries.”

Check the user’s side. If you have RUM, look at failed checkouts and front-end errors for the same window. Some failures never produce a server-side error at all.

The Engineering Decision

The fix for the immediate problem is whatever the traces point to. The more durable decisions are about measurement:

1. Define SLIs for the journeys users care about. For checkout, a good starting point is the proportion of checkout requests that succeed in under a threshold you choose, measured at the load balancer.
2. Set an SLO and alert on its burn rate. Alert when the error budget is being consumed fast enough to matter, using a long and a short window so the alert is both significant and still current. The Google SRE Workbook’s multiwindow, multi-burn-rate approach is a well-documented starting point.
3. Lead dashboards with the user’s view. Put the SLI, the SLO line, and the golden signals (latency, traffic, errors, saturation) for the journey at the top. Keep CPU, memory, and pod counts below, as explanations.
4. Keep health checks about the pod, and test the journey from outside. Liveness and readiness probes should answer “can this pod do its own work?”, not “is every dependency healthy?”. Test the real path with synthetic journeys instead.

There is a real tradeoff in the last point. A probe that fails whenever a shared dependency is slow can turn a partial outage into a full one: a deep liveness check restarts every pod at once, and a deep readiness check pulls every pod out of rotation together. Testing the real path from outside the service avoids both.

The Outcome

Because this is a constructed example, there is no production result to report. If you make these changes, here is how to tell whether they worked:

• Replay a past incident (or run a game day) and check that the SLO burn alert would have fired before the first support ticket arrived.
• Compare user complaints and SLO alerts over a few weeks. Every user-reported problem should correspond to visible budget burn. If one does not, your SLI is measuring the wrong thing.
• Track how often on-call starts an investigation from the SLO dashboard rather than from a ticket.

Lessons from the Field

• A green dashboard means the machinery is running. It does not mean users are succeeding. Measure both, and label them clearly.
• Measure SLIs at the boundary closest to the user that you control, usually the load balancer or ingress.
• Use percentiles and breakdowns. Averages and global rates hide the users who are having the worst day.
• Treat a gap between edge metrics and application metrics as a signal, not noise.
• Keep cause-side panels. They are how you explain the problem once the user-side signals tell you there is one.