Skip to main content

Secrets rotation policy

Source of truth for every secret the gateway holds: name, scope, storage location, rotation cadence, and procedure. Closes SEC-09 from the launch-readiness audit.

Automation reminder: .github/workflows/secret-rotation-reminders.yml runs monthly, reads .github/secrets-inventory.yml, and opens a secret rotation due: <name> issue labelled ops / security whenever a secret's last_rotated plus its interval_days is within 30 days of today. The inventory file is the canonical cadence list — when you rotate a secret, bump last_rotated there.

Scope note. This runbook documents rotation. It does not document provisioning (that lives in infra/terraform/modules/secrets/main.tf) or access-grant patterns (in infra/terraform/modules/iam/). If a rotation procedure produces a new secret resource, provisioning happens via Terraform and rotation then takes over from there.

Inventory​

All secrets live in GCP Secret Manager under project pinpoint-gateway unless otherwise noted. Keys are CMEK-encrypted (var.kms_key_id). Staging and production share Secret Manager but use distinct secret_id values where listed.

Secret secret_idScopeStorageCadenceRotation procedure
gateway-database-urlProductionSecret ManagerOn-demand onlyDB URL
gateway-database-url-stagingStagingSecret ManagerOn-demand onlyDB URL
gateway-internal-api-shared-secretShared (cross-service)Secret Manager, autogen via random_password180 daysInternal shared secret
oauth-signing-keyShared (auth service issues, public services verify via JWK)Secret Manager90 daysOAuth signing key
transit-developer-idShared (TransIT identity)Secret ManagerTSYS-driven (see procedure)TransIT developer ID
xtms-app-idShared (NexGO device mgmt)Secret ManagerNexGO-driven / annualXTMS app credentials
xtms-app-keyShared (NexGO device mgmt)Secret ManagerNexGO-driven / annualXTMS app credentials
apple-pay-platform-integrator-idShared (Apple Pay identity)Secret ManagerApple-driven / annualApple Pay identity cert
apple-pay-integrator-identity-certificateSharedSecret Manager (uploaded out-of-band via gcloud)25 months (Apple-driven)Apple Pay identity cert
apple-pay-integrator-identity-certificate-keySharedSecret Manager (out-of-band)matches identity certApple Pay identity cert
apple-pay-payment-processing-certificateSharedSecret Manager (out-of-band)25 months (Apple-driven)Apple Pay payment processing cert
apple-pay-payment-processing-keySharedSecret Manager (out-of-band)matches processing certApple Pay payment processing cert
checkout-session-secretProduction (HMAC)Secret Manager, autogen via random_password180 daysCheckout session HMAC
checkout-session-secret-stagingStaging (HMAC)Secret Manager, autogen via random_password180 daysCheckout session HMAC
Firebase Admin SDK service account keyShared (if used by management for Firebase Auth verification)Secret Manager as a GCP service account JSON key, OR Workload Identity Federation (WIF) with no persistent key material90 days if JSON key; n/a with WIFFirebase Admin SDK
GCP service account keys (deploy pipelines)CI / CDGitHub Actions secrets via WIF (WIF_PROVIDER, WIF_SERVICE_ACCOUNT)WIF → no key material to rotate; underlying SA impersonation policy reviewed annuallyDeploy SA keys
Internal service-to-service ID tokensEphemeral (issued per-request)Not storedn/a — short-lived, issued by GCP IAMInternal ID tokens

"On-demand only" means rotation is driven by an external event (breach, compromise, Spanner instance rename) rather than a calendar; the reminders workflow still tracks the secret but does not open an issue for it unless the interval_days is set.

Per-secret procedures​

DB URL (Spanner connection string)​

The gateway-database-url and gateway-database-url-staging values are plain JDBC connection strings pointing at the Spanner instance; they carry no credentials of their own because Spanner authenticates via GCP IAM (Cloud Run services use their service-account identity, no static DB username/password exists).

  • What rotates: the SA identity and the IAM bindings that grant Spanner access, not the URL itself.
  • When to rotate: Cloud Run service account impersonation policy is reviewed annually, or on an incident where an SA is suspected of compromise.
  • Procedure:
    1. In infra/terraform/modules/iam/ confirm the list of SAs with roles/spanner.databaseUser. Remove any SA no longer owned by an active Cloud Run service.
    2. If rotating an SA (not the URL): create a new SA in Terraform, grant roles/spanner.databaseUser, update the Cloud Run service to use it, then remove the old binding after one week of overlap.
    3. The URL itself only changes if the Spanner instance is renamed — a rare event, typically tied to a region migration or a DR restore (see Spanner DR runbook).

Internal API shared secret​

Autogenerated by Terraform (random_password.internal_api_shared_secret). Used by internal service clients as a secondary signal alongside Cloud Run IAM — defense in depth, not the primary auth layer.

  • Cadence: 180 days.
  • Procedure:
    1. Run terraform -chdir=infra/terraform taint 'module.secrets.random_password.internal_api_shared_secret' then terraform apply. Terraform generates a new value and creates a new Secret Manager version.
    2. The Cloud Run services that mount this secret read latest by default, so the next revision rollout picks up the new value automatically. To force immediate pickup without a full deploy, restart the Cloud Run services: gcloud run services update gateway-<service> --region=us-east1 --update-env-vars=ROTATION_BUMP=$(date +%s) (any env-var change forces a revision roll).
    3. Bump last_rotated in .github/secrets-inventory.yml.

OAuth signing key​

RSA key used by the auth service to sign OAuth2 access tokens and ID tokens; verified by public-facing services via the JWK endpoint. Rotation is the highest-risk operation on this list because misplaying it invalidates every in-flight token.

  • Cadence: 90 days.
  • Overlap window: the new key must be active for verification before it starts being used for signing. Never flip both in one step.
  • Procedure:
    1. Generate a new 2048-bit RSA key locally: 
      openssl genrsa -out /tmp/oauth-signing-key-$(date +%Y%m%d).pem 2048
    2. Upload as a new version of oauth-signing-key: 
      gcloud secrets versions add oauth-signing-key \
        --project=pinpoint-gateway \
        --data-file=/tmp/oauth-signing-key-$(date +%Y%m%d).pem
    3. The auth service loads all enabled versions of the secret and publishes every public key to the JWK endpoint. Confirm both the old and the new key are present: 
      curl -s https://auth.pinpointgateway.com/.well-known/jwks.json | jq '.keys | length'
      Expected: 2. If the auth service only publishes one key, it has not yet reloaded — roll a new revision to force pickup.
    4. Verification overlap window: leave the new key at "verify only" for 24 hours. Public services cache JWK responses; 24h ensures every instance has refreshed at least once.
    5. Flip the auth service to sign with the new key. This is controlled by the OAUTH_SIGNING_KEY_VERSION env var on the auth Cloud Run service: 
      gcloud run services update gateway-auth \
        --project=pinpoint-gateway \
        --region=us-east1 \
        --update-env-vars=OAUTH_SIGNING_KEY_VERSION=$(gcloud secrets versions list oauth-signing-key --project=pinpoint-gateway --limit=1 --format='value(name)')
    6. Retire the old key after 24 more hours (so any access token issued in the last 60 minutes before the flip has expired). Disable the old Secret Manager version: 
      gcloud secrets versions disable <OLD_VERSION_NUMBER> \
        --secret=oauth-signing-key \
        --project=pinpoint-gateway
    7. Bump last_rotated in .github/secrets-inventory.yml.

If any step fails mid-rotation, do not delete the old key. The only safe recovery is to flip OAUTH_SIGNING_KEY_VERSION back to the old version number and investigate.

TransIT developer ID​

TSYS-assigned identifier tied to the TransIT merchant. The dev cert expires on TSYS's calendar (typically annually for dev, per-merchant for production). TSYS emails a renewal notice ~60 days before expiry.

  • Cadence: TSYS-driven (typically annual). Reminder workflow treats it as 365 days for alerting purposes.
  • Procedure:
    1. When the renewal email arrives from Global Payments, follow their cert-renewal flow in the TSYS merchant portal.
    2. Download the new cert bundle.
    3. Upload as a new version in Secret Manager: 
      gcloud secrets versions add transit-developer-id \
        --project=pinpoint-gateway \
        --data-file=/path/to/new/bundle
    4. Cloud Run services mounting transit-developer-id pick up the latest version on next revision roll. Force a roll by restarting the processing service.
    5. Run a smoke TransIT call via the cert runner (./tools/certification/run-all-tabs.sh --tabs ecommerce --skip-setup) to verify the new cert is accepted upstream.
    6. Retire the old version only after a clean smoke run.
    7. Bump last_rotated in .github/secrets-inventory.yml.

XTMS app credentials​

NexGO XTMS app ID and app key, used for device provisioning. NexGO issues these per integrator account; rotation is rare and annual unless NexGO rotates them on their end.

  • Cadence: annual (calendar-driven), or on NexGO's notice.
  • Procedure:
    1. In the NexGO XTMS portal, regenerate the app key (the app ID is stable; only the key rotates).
    2. Upload the new key: 
      gcloud secrets versions add xtms-app-key \
        --project=pinpoint-gateway \
        --data-file=/tmp/xtms-app-key-$(date +%Y%m%d).txt
    3. Restart the device-provisioning service (or the service that owns XTMS calls — per CLAUDE.md this is handled inside the management service via a client). Monitor logs for XTMS request failures over the next hour.
    4. Bump last_rotated in .github/secrets-inventory.yml.

Apple Pay platform integrator identity certificate​

This is the identity cert Apple issues when you register as a Platform Integrator. It proves that the Peak Gateway platform is who it says it is to Apple Pay infrastructure.

  • Cadence: 25 months (Apple issues these with a 25-month validity; plan rotation at 24 months).
  • Consequence of missing rotation: merchants stop being able to enroll new Apple Pay domain associations. Existing tokenization is not immediately broken, but net new merchant onboarding to Apple Pay halts.
  • Procedure:
    1. At T-60 days: generate a new CSR: 
      openssl req -new -newkey rsa:2048 -nodes \
        -keyout apple-pay-integrator-identity.key \
        -out apple-pay-integrator-identity.csr \
        -subj "/CN=PeakGateway Platform Integrator/O=Peak Gateway/C=US"
    2. Submit the CSR to Apple via the Apple Pay Platform Integrator portal. Wait for Apple to sign and return the cert.
    3. Upload both halves as new versions: 
      gcloud secrets versions add apple-pay-integrator-identity-certificate \
        --project=pinpoint-gateway --data-file=apple-pay-integrator-identity.pem
      gcloud secrets versions add apple-pay-integrator-identity-certificate-key \
        --project=pinpoint-gateway --data-file=apple-pay-integrator-identity.key
    4. Force a roll on services that mount these secrets (processing + online-txn).
    5. Verify by running the Apple Pay tab of the certification suite against the staging merchant: ./tools/certification/run-all-tabs.sh --tabs in-app --skip-setup.
    6. Retire old versions only after a successful Apple Pay session in staging.
    7. Bump last_rotated in .github/secrets-inventory.yml.

Apple Pay payment processing certificate​

This is the per-merchant-class cert that decrypts Apple Pay payment tokens. Missing this rotation breaks production tokenization — every Apple Pay transaction fails until the new cert is in place.

  • Cadence: 25 months (Apple-issued). Target rotation at 24 months.
  • Procedure:
    1. Follow the same CSR flow as the identity cert but through the Apple Pay Payment Processing Certificate section of the portal.
    2. Upload both halves: 
      gcloud secrets versions add apple-pay-payment-processing-certificate \
        --project=pinpoint-gateway --data-file=apple-pay-processing.pem
      gcloud secrets versions add apple-pay-payment-processing-key \
        --project=pinpoint-gateway --data-file=apple-pay-processing.key
    3. Critical: Apple Pay payment decryption supports a rollover window — tokens encrypted under the old cert can still be decrypted as long as the old Secret Manager version remains enabled. Keep the old version enabled for at least 14 days after the new cert goes live (Apple Pay tokens have a time window before expiry).
    4. Verify end-to-end: run a real Apple Pay sandbox transaction through the staging checkout and confirm the processing service decrypts the token successfully.
    5. Bump last_rotated in .github/secrets-inventory.yml.

If decryption starts failing after the new cert is deployed: re-enable the old version immediately; investigate before retrying.

Checkout session HMAC​

HMAC secret used to sign checkout-session tokens issued by the online-txn service. Autogenerated by Terraform (random_password.checkout_session_secret). Rotation needs a dual-key rollover so sessions issued under the old key still validate during cutover.

  • Cadence: 180 days.
  • Expected session TTL: checkout sessions are short (~60 min), so the overlap window is modest.
  • Procedure:
    1. Add a new version: 
      terraform -chdir=infra/terraform taint 'module.secrets.random_password.checkout_session_secret'
      terraform -chdir=infra/terraform apply
    2. The online-txn service must be able to verify signatures using either the old or the new key, while issuing new sessions with the old key for the first overlap period, then flipping to the new key. The service reads both CHECKOUT_SESSION_SECRET_CURRENT and CHECKOUT_SESSION_SECRET_PREVIOUS env vars.
    3. During rotation:
      • Step A: deploy with CURRENT=old, PREVIOUS=new. Both validate; issuance uses old.
      • Step B (2 hours later): flip to CURRENT=new, PREVIOUS=old. Both validate; issuance uses new.
      • Step C (2 hours after B — long enough for any session issued under old to have expired): drop PREVIOUS.
    4. After step C, disable the old Secret Manager version.
    5. Bump last_rotated in .github/secrets-inventory.yml.

Staging (checkout-session-secret-staging) follows the same procedure but can be rotated without overlap during off-peak hours.

Firebase Admin SDK key​

Used by the management service to verify Firebase-issued ID tokens from the admin portal. Two deployment modes:

  • Workload Identity Federation (preferred): the management Cloud Run service's SA is granted roles/firebase.admin on the Firebase project. No persistent key material — nothing to rotate. Reminder workflow does not track this case.
  • JSON key (legacy / local dev only): if the service mounts a Firebase SA JSON key, that key must rotate every 90 days.
  • Procedure (JSON key only):
    1. In the GCP console → IAM → Service Accounts → <firebase-admin-sa>@... → Keys → Create new key (JSON).
    2. Upload as a new secret version: 
      gcloud secrets versions add firebase-admin-sdk-key \
        --project=pinpoint-gateway --data-file=/tmp/firebase-sa.json
    3. Roll the management service to pick it up.
    4. In IAM, delete the old key (not just disable it — SA keys are expensive attack surface).
    5. Bump last_rotated in .github/secrets-inventory.yml.

Deploy pipeline service account keys​

GitHub Actions authenticates to GCP via Workload Identity Federation (WIF_PROVIDER + WIF_SERVICE_ACCOUNT GitHub secrets). No persistent key material exists. Rotation consists of:

  • Annual review of the WIF pool attribute mappings and the list of SAs GitHub is allowed to impersonate.
  • Immediate action if a workflow is exfiltrated or a third party gains write access to the repo: revoke the WIF binding on the target SA in Terraform and re-apply.

No per-secret last_rotated tracking.

Internal service ID tokens​

The InternalServiceClient in libs/security fetches a Google IAM ID token for each outbound service-to-service call. Tokens are signed by Google, valid for 1 hour, cached in-process.

  • Rotation: automatic. Google reissues on expiry.
  • What can break: if ID token exchange starts failing (401 Unauthorized on service-to-service calls):
    1. Check the calling service's SA has roles/run.invoker on the target service in infra/terraform/modules/iam/.
    2. Confirm the target service's Cloud Run config requires authentication (--no-allow-unauthenticated).
    3. Check the GCP metadata server is reachable from the Cloud Run instance (curl to http://metadata.google.internal).
    4. If all of the above are green but tokens are still rejected, file a GCP support ticket — the IAM token service is degraded.

Automation details​

.github/workflows/secret-rotation-reminders.yml reads .github/secrets-inventory.yml every month on the first at 14:00 UTC, computes last_rotated + interval_days, and opens an issue for each secret whose rotation window is within 30 days of today (or is already overdue).

Inventory schema:

secrets:
  - name: oauth-signing-key
    interval_days: 90
    last_rotated: "2026-01-15"
    owner: security-team
    runbook: secrets-rotation#oauth-signing-key

When a secret is rotated, the only bookkeeping required is bumping last_rotated in the inventory and closing the corresponding issue. The workflow is idempotent: if an open issue titled secret rotation due: <name> already exists, it does not open a duplicate.

To adjust cadence, edit interval_days. To retire a secret, remove it from secrets-inventory.yml; the workflow will stop tracking it the following month.