SLO error budgets
Gateway services ship under a 99.9% availability SLO over a rolling 30-day window. That leaves a 0.1% error budget — approximately 43 minutes 12 seconds of allowed bad time per service per 30 days. This doc explains what the budget means, which alerts fire, what happens when the budget runs out, and how resets work.
Terraform source for alerts:
infra/tf/gcp/modules/monitoring/slo_budget.tf.
What the budget measures
For every service in the services map passed to the monitoring module (auth, processing, management, online_txn, merchant_onboarding, status), Cloud Monitoring tracks a single availability SLI against its Cloud Run service. The SLI uses the basic_sli { availability {} } signal — any request with a 5xx response (or a transport-level failure) counts as bad.
The budget is 1 − 0.999 = 0.001 (0.1%) of the rolling-window traffic volume.
| SLO goal | Budget per 30 days (at 100% uptime as baseline) |
|---|---|
| 99.9% | ~43m 12s |
| 99.95% | ~21m 36s |
| 99.99% | ~4m 19s |
The budget recovers continuously: as older-than-30-days bad minutes fall out of the rolling window, remaining budget rises again. So a bad incident doesn't trap a service in a permanent feature-freeze — it just forces a cooldown until the old errors age out.
Alert severity and delivery
Three alert policies watch each service's SLO. They differ in urgency and notification delivery:
| Alert | Condition | Response class | Delivery | Use for |
|---|---|---|---|---|
| Fast burn | Burn rate > 10× over 5 min | CRITICAL | Slack #gateway-alerts + managed email fallback | Active incident — respond immediately |
| Slow burn | Burn rate > 2× over 1 hour | CRITICAL | Slack #gateway-alerts + managed email fallback | Persistent issue — respond promptly |
| Error budget < 10% remaining | Budget fraction < 0.1 for 15 min sustained | WARNING | Cloud Monitoring only by default | Feature-freeze trigger — act during business hours |
The budget-remaining alert uses a 15-minute sustain to avoid reacting to a
transient dip, auto-closes after 7 days if the budget recovers, and routes to
the notification-channel IDs explicitly configured in
var.slo_budget_alert_channels.
Alert routing
The root Terraform module currently leaves var.slo_budget_alert_channels
empty, so the budget-remaining warning is visible in Cloud Monitoring but sends
no outbound notification by default. Critical alerts use Slack
#gateway-alerts with managed email as a fallback. None of these channels
provides automated paging, acknowledgement, or escalation.
Consequences of exhaustion
When the remaining error budget drops below 10%, the service enters feature-freeze until one of:
- The budget recovers above 20% through natural rolling-window drift.
- The next quarterly budget reset (see below) explicitly restores it.
Feature-freeze rules:
- No new features merge into the affected service. This is enforced by convention — the platform team reviews PRs against the frozen service and applies the
freeze-waiverlabel only for changes that measurably reduce burn rate (bug fixes, error-handling improvements, rollbacks, retry-budget reductions). - Existing migrations, deploys, and config changes still proceed if they're reliability-neutral or reliability-positive. A strict freeze would slow down the very work needed to escape the freeze.
- Customer-impacting patches always merge, even if they technically add behaviour — weigh customer risk against SLO compliance on a case-by-case basis.
Exit criteria:
- Budget returns above 20% and no active incident.
- OR 14 days have elapsed since entering freeze with no further burn.
- OR the next quarterly reset lands.
In all cases the platform team documents the exit in the service's monthly reliability review.
Quarterly resets
Every fiscal quarter (1 Jan, 1 Apr, 1 Jul, 1 Oct), the platform team reviews each service's SLO against observed performance and explicitly resets the budget. The reset does three things:
- Adjusts the SLO goal if systematically over- or under-performing. If a service has spent two straight quarters with < 40% budget used it's a signal the SLO is too loose; if it's spent two straight quarters in feature-freeze the SLO is too tight. Tightening increases response load and must be announced one quarter ahead.
- Reconciles the burn-rate alert thresholds so they match the new SLO goal. Fast-burn is always 10× over 5 min and slow-burn is always 2× over 1 hour, but the absolute bad-minute count these resolve to shifts with the goal.
- Lifts any active feature-freezes, regardless of recovery state. The reset is an explicit platform-team call that it's safe to ship again; the platform team carries the risk of resuming at elevated burn until the rolling window catches up.
Reset cadence is documented in infra/tf/gcp/modules/monitoring/slo_budget.tf; current burn rates, remaining budget, and time-to-exhaust projections are surfaced in the SLO burn dashboard defined in infra/tf/gcp/modules/monitoring/dashboards.tf.
Manual response
The budget-remaining warning is not an after-hours wake-up signal. If a team member sees it, they acknowledge it in Slack, apply the feature-freeze rules, and contact the platform engineering lead manually if another owner is needed. There is no automated acknowledgement timer or escalation path.
After resolution, open a reliability post-mortem issue in
pinpointpos/gateway labelled post-mortem + slo-budget. The post-mortem
template is in .github/ISSUE_TEMPLATE/post-mortem.md.
Diagnosing high burn
- Check whether the fast-burn or slow-burn alerts are also firing. If yes, there's probably an active incident — start the relevant incident-response runbook, such as Pub/Sub DLQ drain for consumer-side burn or Spanner disaster recovery for data-plane burn. If no, the burn is likely accumulated from many small incidents over the rolling window.
- Pull recent error logs for the affected service and correlate with deploy timestamps. Error spikes aligned with a deploy point to a regression — roll back first, diagnose later.
- Compare current budget consumption against the historical trend. A step change in burn indicates a regression; a gradual upward drift indicates the SLO target is too tight for the current traffic pattern.
Related docs
- Distributed tracing — service-to-service latency and error correlation.
- Pub/Sub DLQ drain — dead-letter-queue failures, a common source of async error budget burn.
infra/tf/gcp/modules/monitoring/main.tf— burn-rate alert definitions + SLO goals (per service).infra/tf/gcp/modules/monitoring/slo_budget.tf— this doc's implementation source.