← Back to the library
OperationsApplied · 8 min

On-call ignores pages because most alerts are noise

“The pager fires every night for high CPU, a single failed health check, or a disk at 70%. People ack without looking, and the real incident was reported by a customer.”

THE PRINCIPLE

Every page should be urgent, actionable, and need human judgment. If the response is always the same, automate it; if it can wait until morning, make it a ticket.

FIRST MOVES

  1. Export last month's pages and tag each one: acted on, ignored, or duplicate. Delete or demote any alert that nobody acted on.
  2. Define SLIs on the four golden signals at the edge (error ratio, latency of successful requests) and set an SLO for each critical user journey.
  3. Replace static threshold pages with multiwindow burn-rate alerts: page at 14.4x over 1h and 5m or 6x over 6h and 30m, ticket at 1x over 3 days and 6h (for a 99.9% SLO).
  4. Keep cause alerts (disk filling, certificate expiring) only when exhaustion is definite and near, and predict it (predict_linear) rather than using a fixed percentage.
  5. Every alert carries a summary, its impact, and a runbook or dashboard link. Group and inhibit downstream alerts in Alertmanager.

TOOLS: THEN → NOW

Static thresholds on CPU, memory, and single errorsSLO burn-rate alerts on user-facing SLIs (Sloth, Pyrra, OpenSLO, vendor SLO features)
Every alert pages someoneSeverity routing: page, ticket, or dashboard, with grouping and inhibition

PATTERN SNAPSHOT

# 99.9% SLO => error budget 0.001. Needs recording rules for each window.
- alert: CheckoutErrorBudgetBurn
  expr: |
    (job:slo_errors_per_request:ratio_rate1h{job="checkout"} > (14.4 * 0.001)
     and job:slo_errors_per_request:ratio_rate5m{job="checkout"} > (14.4 * 0.001))
    or
    (job:slo_errors_per_request:ratio_rate6h{job="checkout"} > (6 * 0.001)
     and job:slo_errors_per_request:ratio_rate30m{job="checkout"} > (6 * 0.001))
  labels: { severity: page }
  annotations:
    summary: "Checkout is burning its 30-day error budget fast"
    runbook_url: "https://runbooks.example.com/checkout-slo"

CLOSE THE AI. EXPLAIN THIS.

Why does a burn-rate alert need both a long and a short window, and what goes wrong with only the long one?

HOW IT WORKS UNDERNEATH

SOURCES

Guide reviewed