The collection

All the stories in the light

8 true stories of observability done right. Filter by the kind of win, or read them all and borrow the practices behind them.

The Blameless Postmortem That Changed Everything

an engineering manager Jun 20, 2026

An outage could have ended with someone blamed and nothing learned. Instead a blameless postmortem turned it into three concrete fixes and a team that started surfacing risks early.

read the story →

#postmortem #culture #incident-response

The Canary That Caught It

release engineering Jun 13, 2026

A bad deploy raised the error rate the instant it touched real traffic. The canary saw it at one percent, rolled itself back automatically, and 99 percent of users never noticed.

read the story →

#rollout #canary #automation

The Runbook That Worked

the new hire, still calm Jun 6, 2026

A new hire's first night page, resolved in ten calm minutes, because the alert linked straight to a runbook that named the dashboard, the likely cause, and the exact command to run.

read the story →

#runbooks #on-call #incident-response

The Label We Did Not Add

you are welcome, future us May 30, 2026

A pull request tried to add a user ID to a metric. A linter flagged it, a reviewer explained why, and an eight million series cardinality explosion quietly never happened.

read the story →

#metrics #cardinality #culture

Tail Sampling Caught the One in Two Thousand

observability lead May 23, 2026

A bug that corrupted one order in two thousand would have been invisible under head sampling. Tail-based sampling kept every failing trace, and the root cause was found in minutes.

read the story →

#tracing #sampling #latency

The Page That Never Came

the on-call, and grateful for it May 16, 2026

A week on call with zero pages. Not because nothing happened, but because error budgets, symptom-based alerts, and ruthless noise pruning meant only the things worth waking for ever woke anyone.

read the story →

#on-call #alerting #error-budget

The Dashboard That Told the Truth

platform team May 9, 2026

When latency crept up, one dashboard showed the real story in seconds: a data freshness badge, SLO burn lines, and the one graph that mattered. The fix took longer to deploy than to find.

read the story →

#dashboards #golden-signals #slo

The Alert That Saved the Weekend

SRE, payments May 2, 2026

A single well-tuned alert on a slow memory climb fired on Friday afternoon, days before it would have become a Saturday-night outage. Nobody lost a weekend.

read the story →

#alerting #slo #capacity