Skip to main content

Gift card ledger integrity

Owner: processing service on-call Alert path: scheduled integrity sweep email plus ops dashboard metrics

What this is​

The gift card ledger (gift_card_transactions) is append-only. Two invariants must hold across every card:

  1. The sum of all ledger row amounts for a card equals gift_cards.current_balance_cents.
  2. Each ledger row's balance_after_cents equals the previous row's balance plus that row's amount_cents. The first row's balance equals its amount.

The nightly GiftCardLedgerIntegrityCheck scheduler walks active gift cards and verifies both invariants. Any violation means corruption: either application code regressed past the repository chain-consistency check, or a direct data write bypassed the repository.

Symptoms​

  • Email alert: Gift card ledger integrity alarm
  • Counter gift_card_ledger_integrity_violations_total{type=sum|chain} is above zero
  • Gauge gift_card_ledger_integrity_last_violations reflects the most recent run
  • Scheduler endpoint POST /api/v1/internal/scheduled/gift-card-ledger-integrity-sweep returns 5xx with a LedgerIntegrityException payload

Triage​

  1. Identify the affected cards from the alert payload's cardIds field or the most recent integrity sweep error in Cloud Logging.
  2. Distinguish sum violations from chain violations:
    • A sum-only violation means current_balance_cents was mutated outside the normal issue, redeem, reload, refund, expire, revoke, or adjust paths.
    • A chain violation means a ledger row was inserted with an inconsistent balance_after_cents, usually from a repository regression or a backfill that bypassed the repository.
  3. Quarantine each affected card with POST /management/api/v1/locations/{locationId}/gift-cards/{id}/mark-lost so it cannot be redeemed while the incident is investigated.

Root Cause​

For each affected card, pull the full ledger in chronological order:

SELECT id, type, amount_cents, balance_after_cents, created_at, actor_type, actor_id, reason
  FROM gift_card_transactions
 WHERE gift_card_id = ?
 ORDER BY created_at ASC;

Then pull the persisted balance:

SELECT current_balance_cents, version, status, updated_at
  FROM gift_cards
 WHERE id = ?;

Reconstruct the running sum from the ledger and compare it with current_balance_cents. The first row where the running sum diverges is the corruption point. Cross-reference that row's created_at with deploys, manual database sessions, and Cloud Audit logs.

Remediation​

The ledger is the source of truth unless investigation proves that a ledger row itself was missed or incorrectly written. Restore consistency by appending one audited adjustment through the platform-admin adjust endpoint.

MANAGEMENT_URL="https://api.peakgateway.co/management"
LOCATION_ID="<location id>"

curl -X POST \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"amountCents": -123, "reason": "INC-XXXX integrity reconciliation"}' \
  "${MANAGEMENT_URL}/api/v1/locations/${LOCATION_ID}/gift-cards/${CARD_ID}/adjust"

After remediation, re-run the integrity sweep:

PROJECT="pinpoint-gateway"
REGION="us-east1"
PROCESSING_URL="$(gcloud run services describe gateway-processing \
  --project="${PROJECT}" \
  --region="${REGION}" \
  --format='value(status.url)')"

curl -X POST \
  -H "Authorization: Bearer $SCHEDULER_TOKEN" \
  "${PROCESSING_URL}/api/v1/internal/scheduled/gift-card-ledger-integrity-sweep"

The sweep should return 200 with no violations.

Prevention​

  • Never mutate gift_cards or gift_card_transactions outside the repository.
  • Never disable the chain-consistency check in GiftCardTransactionRepository.insert.
  • Add a regression test to GiftCardTransactionRepositoryTest for every new insertion path.
  • For data backfills that need ledger surgery, use repository methods and attach an audit log entry per row.

Related Docs​