Alert runbook index

Every Cloud Monitoring alert policy defined in infra/tf/gcp/modules/monitoring/*.tf has a matching runbook in this directory. When an alert arrives through a configured Slack or managed email channel, find the row below and click through.

The canonical public URL for a runbook is https://support-docs.peakgateway.co/ops/runbooks/<slug>. That URL belongs in the alert policy's documentation.content block so the Cloud Monitoring notification carries the link inline.

Catalog

Alert policy (Terraform resource)	Display name	Severity	Runbook
google_monitoring_alert_policy.slo_fast_burn	<service> SLO fast burn (10x / 5 min)	WARNING (operator-inferred; no explicit severity in TF)	slo-fast-burn
google_monitoring_alert_policy.slo_slow_burn	<service> SLO slow burn (2x / 1 hr)	WARNING (operator-inferred)	slo-slow-burn
google_monitoring_alert_policy.pubsub_dlq_depth	Pub/Sub DLQ has messages: <subscription>	ERROR	pubsub-dlq-depth
google_monitoring_alert_policy.webhook_dlq_depth_persistent	Webhook DLQ depth > 0 for any organization	WARNING	webhook-dlq-depth

Scheduled Integrity Checks

Check	Signal	Runbook
GiftCardLedgerIntegrityCheck	Email alert Gift card ledger integrity alarm plus gift_card_ledger_integrity_* metrics	gift-card-ledger-integrity

Test and CI Runbooks

Area	Runbook
Emulator-backed integration tests	integration-tests

Each runbook covers:

• What the alert means — what is broken and what the user-visible impact is.
• First three diagnostic steps — specific greps, dashboards, and revision checks.
• Rollback criteria — what evidence justifies flipping Cloud Run back to the previous revision.
• Escalation path — who the first responder should contact manually if they cannot resolve the incident.

When a runbook does not exist

If an alert fires and there is no runbook here, treat it as a process bug: file a ticket labelled ops / runbook-gap, snapshot the dashboard, and coordinate the response manually in Slack. There is no automated acknowledgement timer or escalation ladder.

Planned operator drills are not always alert-driven. The canonical example is the Spanner disaster-recovery runbook, which is used for manual restore rehearsal and evidence capture even when no alert is active.

For those manual drills, treat the runbook page itself as procedure only, not as proof of readiness. Closure-grade evidence still means:

• one canonical drill note or issue
• one durable attachment set or artifact directory
• one operator summary saying whether the measured target was met or tracked remediation is required

Cross-service context

• SLO burn-rate alerts use Cloud Monitoring's select_slo_burn_rate function over the per-service availability SLO (google_monitoring_slo.availability, target 0.999 over 30 days). Both fast and slow variants fire against the same SLO resource with different windows.
• The Pub/Sub DLQ-depth alert is keyed per dead-letter subscription. For message-triage procedure (replay vs. quarantine), see the deeper Pub/Sub DLQ drain runbook.
• The webhook DLQ-depth alert is keyed per Organization via the management service's webhook_dlq_depth custom metric and is distinct from the raw Pub/Sub dead-letter alert.
• The Cloud Monitoring policies in the catalog use the configured Slack channel (#gateway-alerts in production) and managed email channels. The separate error-budget warning remains Cloud Monitoring-only unless slo_budget_alert_channels is deliberately configured.
• No current notification channel provides automated paging, acknowledgement, or escalation. incident.io is inactive and is not part of the current response path.