Rollouts Resolved calmly
The Canary That Caught It
A bad deploy raised the error rate the instant it touched real traffic. The canary saw it at one percent, rolled itself back automatically, and 99 percent of users never noticed.
All systems calm · 0 pages tonight
Observability,
Observability,
done right.
Stories of observability that worked: the alert that saved the weekend, the dashboard that told the truth, and the 3am page that never came. Plus the exact practice behind every one.
Series Beyond the Dashboard: the engineering behind the stories →- Metrics
- Logs
- Traces
- SLOs
- Error budgets
- OpenTelemetry
- Kubernetes
- Canary releases
- Tail sampling
- Runbooks
- Golden signals
- Blameless postmortems
- 8stories, each with its practice
- 13practices, explained in plain language
- 19themes, from SLOs to culture
- 0names published without consent
Postmortems, but happy
Incidents that didn't happen
Each story is written up like an incident report with a happy ending, and closes with the practice you can borrow for your own team.
Metrics Resolved calmly
The Label We Did Not Add
A pull request tried to add a user ID to a metric. A linter flagged it, a reviewer explained why, and an eight million series cardinality explosion quietly never happened.
Tracing Resolved calmly
Tail Sampling Caught the One in Two Thousand
A bug that corrupted one order in two thousand would have been invisible under head sampling. Tail-based sampling kept every failing trace, and the root cause was found in minutes.
On-call Resolved calmly
The Page That Never Came
A week on call with zero pages. Not because nothing happened, but because error budgets, symptom-based alerts, and ruthless noise pruning meant only the things worth waking for ever woke anyone.
Dashboards Resolved calmly
The Dashboard That Told the Truth
When latency crept up, one dashboard showed the real story in seconds: a data freshness badge, SLO burn lines, and the one graph that mattered. The fix took longer to deploy than to find.
Alerting Resolved calmly
The Alert That Saved the Weekend
A single well-tuned alert on a slow memory climb fired on Friday afternoon, days before it would have become a Saturday-night outage. Nobody lost a weekend.
An O11y Paradise series
Beyond the Dashboard
Real signals. Real engineering. Beyond the graphs.
The stories show what calm looks like. This series explores the engineering behind it: investigations, architecture decisions, and the tradeoffs that keep systems reliable.
Explore the series →Your Kubernetes Pod Restarted. Now What?
A step-by-step investigation of a restarting container: pod status, termination reasons, events, previous logs, resource metrics, and traces.
Kubernetes and cloud-native · Incident investigation · Logs, metrics, traces, and profiling 6 min read
OpenTelemetry Collects the Data. Who Makes Sense of It?
OpenTelemetry standardizes how telemetry is produced and shipped. Correlation, storage, and investigation are still architecture decisions you have to make on purpose.
OpenTelemetry · Observability architecture · Telemetry pipelines 6 min read
Your Dashboard Is Green. Your Users Are Still Suffering.
Infrastructure health and user experience are different measurements. Why a wall of green panels can sit beside a bad day for users, and how to measure from where users stand.
Reliability engineering · Logs, metrics, traces, and profiling · Incident investigation 6 min read
Borrow the calm
The practices behind the calm
Plain-language answers to the questions every on-call engineer eventually asks.
What is symptom-based alerting? Symptom-based alerting fires on what users actually feel, such as high latency or error rate, rather than on internal causes like a single restart or CPU spike. Read the answer → What is the difference between head and tail sampling? Sampling keeps only a fraction of traces to control cost. Read the answer → What is a canary release? A canary release sends a new version to a small slice of real traffic first, running it beside the stable version. Read the answer → How do you prevent a cardinality explosion? Every unique combination of label values becomes its own time series, so an unbounded label like a user ID can create millions. Read the answer →
Add to the light
Got a story that let you sleep?
Share the win you are quietly proud of. A human reads every submission, and you choose the byline, including none at all.
- 1Write it. The setup, what you did right, and the moment it paid off.
- 2We review it. Privately. Nothing is published automatically.
- 3It joins the light. For the next engineer who needs the calm.
Editorial standards
Vendor-neutral by design
The practices here work with any stack. Tools are discussed on their merits, contributors disclose their affiliations, and simulated or illustrative examples are labeled as such.