TransIT Certification Run Runbook

Audience: Anyone who needs to re-run the TransIT v6.2 cert suite (Case 00230402) against the sandbox or production merchant credentials.

Tool: tools/certification/ (Kotlin cert runner) + tools/certification/run-all-tabs.sh (orchestration wrapper).

Tracks: Launch-readiness audit TST-05, original issue #237.

Placeholder convention: the commands below contain literal placeholders wrapped in angle brackets (e.g. <from gcloud projects list>, <timestamp>, <Greg's email at Global Payments>, <YYYY-MM-DD>). Fill these in at run time --- they are intentional, not TODOs.

What this runbook covers

How to obtain credentials for the cert runner.
How to run a single tab or all three tabs back-to-back.
How to handle common failure modes per step.
How to package evidence and submit to Global Payments.

Red rows are skippable

Greg Crosby at Global Payments marks rows in the cert Excel script red to indicate they do not apply to Pinpoint POS. The tab definitions in tools/certification/src/main/kotlin/com/myriad/gateway/certification/tabs/*.kt already exclude red rows inline via // SKIP (red) comments next to every omission — no additional filter is needed. If a new row appears red in a later spec revision, update the relevant tab .kt file to drop it and add a // SKIP (red) comment next to the removal. The TST-05 plan (docs/superpowers/plans/2026-04-13-tst-05-unstick-237-transit-cert-runner.md) captures the same rule.

Verify the inline markers still exist:

grep -c 'SKIP.*(red)' tools/certification/src/main/kotlin/com/myriad/gateway/certification/tabs/*.kt

Each tab file that has red rows in the spec should report a nonzero count.

Local-only dry-run prep

Use this checklist when preparing the cert runner without live gateway services, external TransIT calls, or Terraform:

# Confirm the wrapper parses options without requiring env or network access.
./tools/certification/run-all-tabs.sh --help

# Build the runner locally. Bazel may use the configured remote build cache,
# but this does not call gateway, management, TransIT, or Terraform.
bazel build //tools/certification:runner

# Run the local MockWebServer-backed tests.
bazel test //tools/certification:cert_client_test \
  //tools/certification:route_policy_test \
  //tools/certification:step_runner_test

# Re-check that red-row omissions are still documented in the tab definitions.
grep -c 'SKIP.*(red)' tools/certification/src/main/kotlin/com/myriad/gateway/certification/tabs/*.kt

Do not run the following during a local-only prep pass:

tools/certification/scripts/check-sandbox-connectivity.sh — curls the public API and management API.
tools/certification/run-all-tabs.sh without --help — runs setup and certification tabs unless you provide real skip/tab options, and the tab runner still submits transaction requests.
bazel run //tools/certification:runner -- setup or -- run — activates locations, imports credentials, creates sessions, and/or sends transaction traffic.
tools/certification/scripts/package-run.sh — uploads to GCS when gsutil is available.

Prerequisites

Staging merchant 888000003647 must be active. The /api/v1/merchants/{id} endpoint lives on the public management surface, so confirm via the public management URL:
```
export CERT_PUBLIC_MANAGEMENT_URL="https://staging-api.peakgateway.co/management"

curl -sS -H "Authorization: Bearer $CERT_API_KEY" \
  -H "X-Merchant-ID: 888000003647" \
  "$CERT_PUBLIC_MANAGEMENT_URL/api/v1/merchants/888000003647"
```
If not, create via the admin portal and run bazel run //tools/certification:runner -- setup ... once. The pre-flight script (tools/certification/scripts/check-sandbox-connectivity.sh) will consume CERT_PUBLIC_MANAGEMENT_URL too when set.
OAuth client for the runner. Credentials in Secret Manager:
- cert-runner-oauth-client-id
- cert-runner-oauth-client-secret
GCS bucket for evidence retention. gs://peak-cirunners-cert-runs/ with the GitHub Actions service account as roles/storage.objectCreator.

Full run (the happy path)

# 1. Export config
export PROJECT_NUMBER=<from gcloud projects list>
export CERT_PUBLIC_API_URL="https://staging-api.peakgateway.co"
export CERT_PUBLIC_MANAGEMENT_URL="$CERT_PUBLIC_API_URL/management"

# 2. Fetch a fresh OAuth token
CLIENT_ID="$(gcloud secrets versions access latest --secret=cert-runner-oauth-client-id)"
CLIENT_SECRET="$(gcloud secrets versions access latest --secret=cert-runner-oauth-client-secret)"
export CERT_API_KEY="$(curl -sS -u "$CLIENT_ID:$CLIENT_SECRET" \
  -d grant_type=client_credentials \
  -d 'scope=session:create txn:process' \
  "https://gateway-auth-staging-${PROJECT_NUMBER}.us-east1.run.app/oauth2/token" \
  | python3 -c 'import sys,json; print(json.load(sys.stdin)["access_token"])')"
export CERT_MGMT_API_KEY="$(curl -sS -u "$CLIENT_ID:$CLIENT_SECRET" \
  -d grant_type=client_credentials \
  -d 'scope=admin:*' \
  "https://gateway-auth-staging-${PROJECT_NUMBER}.us-east1.run.app/oauth2/token" \
  | python3 -c 'import sys,json; print(json.load(sys.stdin)["access_token"])')"
export CERT_ORGANIZATION_ID=<gateway-organization-id>
export CERT_LOCATION_ID=<gateway-location-id>
export CERT_TRANSIT_MID=888000003647
export CERT_TRANSIT_TERMINAL_NUMBER=75959504
export CERT_DEVICE_ID=88800000364701

# 3. Pre-flight
./tools/certification/scripts/check-sandbox-connectivity.sh

# 4. Run the default automated tabs
./tools/certification/run-all-tabs.sh

Expected: the script runs setup + ecommerce + moto + recurring sequentially, each writing to tools/certification/cert-runs/full-<timestamp>/<tab>/. At the end, it generates cert-report.txt, archives the run as a .tar.gz, and uploads to GCS. It only emits a signed URL when CERT_RUN_SIGNURL_KEY is set to a service-account JSON key path; otherwise it prints the gcloud storage sign-url command to run manually.

Total wall clock: ~4 hours (mostly settlement waits).

TSYS review wants one tab per settlement. After each tab settlement, email the tab name, settlement date, and returned batch/settlement number to the TSYS reviewer.

Partial runs

# Single tab
./tools/certification/run-all-tabs.sh --tabs moto

# Resume from a specific failed step
./tools/certification/run-all-tabs.sh --tabs ecommerce --resume-from ec2_5

# Skip setup (already activated)
./tools/certification/run-all-tabs.sh --skip-setup

# Skip the pre-flight (useful when iterating on a fix)
./tools/certification/run-all-tabs.sh --skip-preflight --skip-setup --tabs ecommerce

# Reuse an existing run root
./tools/certification/run-all-tabs.sh --run-root tools/certification/cert-runs/full-<timestamp> \
  --skip-setup --tabs ecommerce --resume-from ec2_5

# Include the Non-SDK In-App tab
./tools/certification/run-all-tabs.sh --include-in-app

Triage: what to do when a step fails

The runner marks the step FAILED and continues to the next step. Inspect the JSON log in the run directory:

cat tools/certification/cert-runs/full-<timestamp>/ecommerce/<step-id>.json

Common patterns and fixes:

Response pattern	Cause	Fix
`"responseStatus": 401`	Token expired	Re-fetch per "Full run" Step 2
`"responseStatus": 403` on webhook endpoints	Token missing scope	Add `webhook:manage` to the public API token scopes
`"responseStatus": 403` on `/management/api/v1/...`	Management token missing admin scope	Mint `CERT_MGMT_API_KEY` with `admin:*` and retry
`"responseStatus": 404` on `/api/v1/transactions/<id>/capture`	Transaction ID not yet persisted in Spanner	Timing issue --- rerun the step with `--resume-from <step-id>`
Status = `DECLINED` when `APPROVED` expected	Possible real TransIT decline --- check response code	If code is `A0000` the card was approved but our status mapping is wrong. Fix in `validateResponse` in `StepRunner.kt`. If code is anything else, escalate to Greg
Status = `APPROVED` when `PARTIAL_APPROVAL` expected	Partial-auth flag not set in request body	Confirm `partialAuthSupport: true` is in `buildSaleBody` for this step
`NullPointerException` in runner	Response shape mismatch --- a field the runner expected was null	Add null-safe extraction in the appropriate `execute*` method
Timeout on session creation	online-txn service cold start	Extend session creation timeout in `CertClient` from 60s to 120s. Staging-only workaround --- INF-10 (staging `min_instances >= 1`) fixes it permanently

Non-SDK In-App Tab

Run the v6.2 Non-SDK In-App card-not-present rows with:

./tools/certification/run-all-tabs.sh --tabs in-app --skip-setup

The tab uses the same public API and management tokens as the other tabs. It does not require a device-specific Apple Pay sandbox setup.

Submission to Global Payments

After a green full run (setup + default tabs + in-app tab if in scope):

./tools/certification/scripts/package-run.sh tools/certification/cert-runs/full-<timestamp>

This uploads the tarball to GCS and prints a signed URL (when CERT_RUN_SIGNURL_KEY points at a service-account JSON key). Email Greg Crosby (<Greg's email at Global Payments>) with:

Subject: Case 00230402 --- Pinpoint POS Cert Submission (run <timestamp>)
Body: a short summary, the signed URL, and a list of any steps that required notes.
Attachment: the filled-in cert Excel spreadsheet (populated by hand from cert-report.txt).

What this runbook covers​

Red rows are skippable​

Local-only dry-run prep​

Prerequisites​

Full run (the happy path)​

Partial runs​

Triage: what to do when a step fails​

Non-SDK In-App Tab​

Submission to Global Payments​