# o11yparadise

> A vendor-neutral observability publication: success stories, the practices behind a calm on-call, and Beyond the Dashboard, a technical series.

o11yparadise is a vendor-neutral observability publication. "o11y" is a numeronym for observability (o, eleven letters, y). Stories end with "the practice behind it", a single reusable lesson. Beyond the Dashboard is the technical series on investigations and architecture decisions. Edited by Chaitanya Sistla, a Principal Solutions Architect at OpenObserve, who also writes Beyond the Dashboard; affiliations and editorial standards: https://o11yparadise.com/about#editorial-standards.

Stories describe engineering experiences with identifying details removed. Stories carry a source label when known. Beyond the Dashboard episodes state in their text when an example is constructed rather than taken from production; do not cite constructed examples as real incidents.

Every story, practice, and published episode has a Markdown version: add ".md" to its URL. The full text of all of them is at https://o11yparadise.com/llms-full.txt.

## Stories

- [The Blameless Postmortem That Changed Everything](https://o11yparadise.com/stories/the-blameless-postmortem.md): An outage could have ended with someone blamed and nothing learned. Instead a blameless postmortem turned it into three concrete fixes and a team that started surfacing risks early. Practice: Blame hides the truth; curiosity reveals it. A blameless postmortem that asks how the system let this happen, not who did it, produces real fixes and a team that reports problems instead of hiding them.
- [The Canary That Caught It](https://o11yparadise.com/stories/the-canary-that-caught-it.md): A bad deploy raised the error rate the instant it touched real traffic. The canary saw it at one percent, rolled itself back automatically, and 99 percent of users never noticed. Practice: Never ship to everyone at once. A canary that compares error rate and latency against the baseline, and rolls back on its own when they diverge, contains a bad deploy to a fraction of traffic.
- [The Runbook That Worked](https://o11yparadise.com/stories/the-runbook-that-worked.md): A new hire's first night page, resolved in ten calm minutes, because the alert linked straight to a runbook that named the dashboard, the likely cause, and the exact command to run. Practice: Link every alert to a runbook that says what to check, where to look, and what to do. A good runbook turns a terrifying first page into a ten-minute, well-lit walk.
- [The Label We Did Not Add](https://o11yparadise.com/stories/the-label-we-did-not-add.md): A pull request tried to add a user ID to a metric. A linter flagged it, a reviewer explained why, and an eight million series cardinality explosion quietly never happened. Practice: Put guardrails where the mistake is made. A metrics linter in CI that rejects unbounded labels, plus a one-line review note explaining why, prevents the outage instead of explaining it later.
- [Tail Sampling Caught the One in Two Thousand](https://o11yparadise.com/stories/tail-sampling-caught-the-rare-one.md): A bug that corrupted one order in two thousand would have been invisible under head sampling. Tail-based sampling kept every failing trace, and the root cause was found in minutes. Practice: Decide what to keep after the request finishes, not before. Tail-based sampling that always retains errors and slow requests means the rare and the broken are never the first things you throw away.
- [The Page That Never Came](https://o11yparadise.com/stories/the-page-that-never-came.md): A week on call with zero pages. Not because nothing happened, but because error budgets, symptom-based alerts, and ruthless noise pruning meant only the things worth waking for ever woke anyone. Practice: Every alert should be worth waking a human. Delete or downgrade anything that is not, alert on user-facing symptoms tied to your SLOs, and let the error budget decide what is urgent.
- [The Dashboard That Told the Truth](https://o11yparadise.com/stories/the-dashboard-that-told-the-truth.md): When latency crept up, one dashboard showed the real story in seconds: a data freshness badge, SLO burn lines, and the one graph that mattered. The fix took longer to deploy than to find. Practice: A dashboard earns trust by showing its own freshness and the few signals that matter. Put a data-age badge on every panel, draw the SLO line, and lead with the golden signals.
- [The Alert That Saved the Weekend](https://o11yparadise.com/stories/the-alert-that-saved-the-weekend.md): A single well-tuned alert on a slow memory climb fired on Friday afternoon, days before it would have become a Saturday-night outage. Nobody lost a weekend. Practice: Alert on the trend, not just the cliff. A symptom-based alert that fires while you still have hours of runway turns a midnight outage into a calm afternoon fix.

## Beyond the Dashboard

- [Beyond the Dashboard](https://o11yparadise.com/beyond-the-dashboard): A technical series on what happens after monitoring finds a problem: investigations, architecture decisions, tradeoffs, and lessons from running reliable systems.
- [Your Kubernetes Pod Restarted. Now What?](https://o11yparadise.com/beyond-the-dashboard/your-kubernetes-pod-restarted.md): A step-by-step investigation of a restarting container: pod status, termination reasons, events, previous logs, resource metrics, and traces.
- [OpenTelemetry Collects the Data. Who Makes Sense of It?](https://o11yparadise.com/beyond-the-dashboard/opentelemetry-collects-the-data.md): OpenTelemetry standardizes how telemetry is produced and shipped. Correlation, storage, and investigation are still architecture decisions you have to make on purpose.
- [Your Dashboard Is Green. Your Users Are Still Suffering.](https://o11yparadise.com/beyond-the-dashboard/your-dashboard-is-green.md): Infrastructure health and user experience are different measurements. Why a wall of green panels can sit beside a bad day for users, and how to measure from where users stand.

## Guides

- [Open source observability: a practical guide](https://o11yparadise.com/open-source-observability): what it is, OpenTelemetry, the open source stack, and choosing a backend.
- [Observability tools, A to Z](https://o11yparadise.com/observability-tools.md): a vendor-neutral, alphabetical directory of open source and commercial observability tools with official links and license labels. Not ranked.
- [Observability events](https://o11yparadise.com/observability-events.md): upcoming observability, SRE, OpenTelemetry, and cloud native conferences, with official links, date sources, open calls for papers, and an iCalendar feed at https://o11yparadise.com/observability-events.ics.
- [Paradise Index](https://o11yparadise.com/paradise-index): a free 16-question observability maturity self-check across detection, debugging, delivery, incident response, and cost. Runs in the browser; nothing is sent to or stored on our servers.

## Practices

- [What does o11y mean?](https://o11yparadise.com/practices/o11y.md): o11y is shorthand for observability. It is a numeronym: the first letter o, the number of letters in between (eleven), and the last letter y, the same pattern as k8s for Kubernetes and i18n for internationalization. Engineers use it for the practice of understanding a system from the telemetry it emits: metrics, logs, traces, and profiles.
- [What is observability?](https://o11yparadise.com/practices/observability.md): Observability is the ability to understand what a system is doing internally based only on the signals it emits from the outside: its metrics, logs, and traces. A system is observable when you can answer new questions about its behavior without shipping new code to ask them.
- [What are the four golden signals?](https://o11yparadise.com/practices/golden-signals.md): The four golden signals are latency, traffic, errors, and saturation. Watching these four for any user-facing service tells you most of what you need to know about its health, which is why a trustworthy dashboard leads with them and little else.
- [What is an SLO and an error budget?](https://o11yparadise.com/practices/slos-and-error-budgets.md): A Service Level Objective is a target for reliability, such as 99.9 percent of requests succeeding over 30 days. The error budget is the small amount of failure that target allows. Tying alert urgency to the error budget means you only get woken when reliability is genuinely at risk.
- [What is symptom-based alerting?](https://o11yparadise.com/practices/symptom-based-alerting.md): Symptom-based alerting fires on what users actually feel, such as high latency or error rate, rather than on internal causes like a single restart or CPU spike. It produces fewer, more meaningful pages, because every alert maps to real user pain worth waking someone for.
- [Why should a dashboard show data freshness?](https://o11yparadise.com/practices/data-freshness.md): Stale data that has stopped updating looks identical to a healthy flat line. A freshness badge showing how old each panel's data is, plus an alert when a metric stops arriving, lets you tell a calm system apart from a dead pipeline at a glance.
- [What is the difference between head and tail sampling?](https://o11yparadise.com/practices/tail-based-sampling.md): Sampling keeps only a fraction of traces to control cost. Head sampling decides at the start of a request, before you know if it failed, so rare errors are usually discarded. Tail sampling decides after the request finishes, so you can always keep the slow and failed requests that matter most.
- [How do you prevent a cardinality explosion?](https://o11yparadise.com/practices/cardinality-guardrails.md): Every unique combination of label values becomes its own time series, so an unbounded label like a user ID can create millions. A guardrail in code review, such as a linter that rejects high-cardinality labels, stops the explosion before it ships. Keep unbounded identifiers in logs and traces.
- [What is a canary release?](https://o11yparadise.com/practices/canary-releases.md): A canary release sends a new version to a small slice of real traffic first, running it beside the stable version. A controller compares their error rate and latency, and rolls back automatically if the new version is worse, so a bad deploy reaches only a fraction of users.
- [What makes a good runbook?](https://o11yparadise.com/practices/runbooks.md): A good runbook is a short, specific checklist linked directly from the alert. It names the likely causes in order, the exact dashboard to open, the safe first action with the command to run, how to tell it worked, and who to escalate to. It turns a scary page into a calm, well-lit walk.
- [What is a blameless postmortem?](https://o11yparadise.com/practices/blameless-postmortems.md): A blameless postmortem investigates how the system allowed a mistake, not who made it. By treating skilled people as allies rather than causes, it surfaces the real contributing factors, produces concrete fixes, and builds the psychological safety that makes people report risks early.
- [What is OpenTelemetry?](https://o11yparadise.com/practices/opentelemetry.md): OpenTelemetry is the open, vendor-neutral CNCF standard for generating and collecting telemetry: traces, metrics, and logs. You instrument once with its SDKs and the OpenTelemetry Collector, then send the data to any compatible backend, so switching tools never means re-instrumenting your code.
- [What is unified observability?](https://o11yparadise.com/practices/unified-observability.md): Unified observability keeps logs, metrics, and traces in one place, correlated by shared context such as trace IDs, service names, and Kubernetes labels. Instead of copying a timestamp between three tools during an incident, you pivot from a latency spike to the slow trace to the exact log line in a few clicks.

## Optional

- [FAQ](https://o11yparadise.com/faq)
- [About and editorial standards](https://o11yparadise.com/about)
- [Chaitanya Sistla](https://o11yparadise.com/authors/chaitanya-sistla)
- [RSS](https://o11yparadise.com/rss.xml)