TransIT Certification Run Runbook

Audience: Anyone who needs to re-run the TransIT v6.2 cert suite (Case 00230402) against the sandbox or production merchant credentials.

Tool: tools/certification/ (Kotlin cert runner) + tools/certification/run-all-tabs.sh (orchestration wrapper).

Placeholder convention: the commands below contain literal placeholders wrapped in angle brackets (e.g. <from gcloud projects list>, <timestamp>, <Greg's email at Global Payments>, <YYYY-MM-DD>). Fill these in at run time --- they are intentional, not TODOs.

What this runbook covers

How to obtain credentials for the cert runner.
How to run a single tab or all three tabs back-to-back.
How to handle common failure modes per step.
How to package evidence and submit to Global Payments.

The In-App tab remains in the runner for scope-dependent certification work, but it is not part of the default automated sequence. Run it explicitly with --tabs in-app or CERT_TABS=in-app only when TSYS confirms it is in scope.

EPX and EPG scope

This runbook's automated workflow is production-path-only. Retained Gateway rows use only these Gateway-owned routes:

Checkout and wallet rows use POST /api/v1/checkout/sessions, POST /api/v1/checkout/sessions/{sessionId}/pay, and GET /api/v1/checkout/sessions/{sessionId}; Apple Pay data is entered on the operator device.
Sale/auth rows use POST /api/v1/virtual-terminal/sale and POST /api/v1/virtual-terminal/auth.
Transaction lifecycle and inquiry rows use POST /api/v1/transactions/{transactionId}/capture, POST /api/v1/transactions/{transactionId}/void, POST /api/v1/transactions/{transactionId}/refund, POST /api/v1/transactions/{transactionId}/tip-adjust, POST /api/v1/transactions/{transactionId}/enrich, GET /api/v1/transactions/{transactionId}, and POST /api/v1/virtual-terminal/force-auth.
SetupIntent rows use POST /api/v1/setup-intents, GET /api/v1/setup-intents/{id}, and POST /api/v1/setup-intents/{id}/confirm.
EPX ACH prenotes use POST /api/v1/organizations/{organizationId}/locations/{locationId}/ach-prenotes.

Batch-close rows are manual/operator checkpoints. The runner sends no Gateway batch-close request, preserves settlement rows, and requires signed external confirmation before the exact checkpoint resume. BRIC and EPG order rows, standalone provider-direct shopper/card-verification rows, provider-direct wallet-wire rows, and EPX/EPG batch certification are deferred and have no runner rows. EPG SetupIntent rows still cover Gateway-owned verification, shopper, and stored-card flows; EPG wallet rows use hosted Gateway checkout rather than provider wire. The EPG registry remains certified: false.

Red rows are skippable

Greg Crosby at Global Payments marks rows in the cert Excel script red to indicate they do not apply to Pinpoint POS. The tab definitions in tools/certification/src/main/kotlin/com/myriad/gateway/certification/tabs/*.kt already exclude red rows inline via // SKIP (red) comments next to every omission; no additional filter is needed. When a spec revision marks an additional row red, update the relevant tab .kt file to drop it and add a // SKIP (red) comment next to the removal.

Verify the inline markers still exist:

grep -c 'SKIP.*(red)' tools/certification/src/main/kotlin/com/myriad/gateway/certification/tabs/*.kt

Each tab file that has red rows in the spec should report a nonzero count.

Local-only dry-run prep

Use this checklist when preparing the cert runner without live gateway services, external TransIT calls, or Terraform:

# Confirm the wrapper parses options without requiring env or network access.
./tools/certification/run-all-tabs.sh --help

# Build the runner locally. Bazel may use the configured remote build cache,
# but this does not call gateway, management, TransIT, or Terraform.
bazel build //tools/certification:runner

# Run the local MockWebServer-backed tests.
bazel test //tools/certification:cert_client_test \
  //tools/certification:route_policy_test \
  //tools/certification:step_runner_test

# Re-check that red-row omissions are still documented in the tab definitions.
grep -c 'SKIP.*(red)' tools/certification/src/main/kotlin/com/myriad/gateway/certification/tabs/*.kt

Do not run the following during a local-only prep pass:

tools/certification/run-all-tabs.sh without --help — runs setup and certification tabs unless you provide real skip/tab options, and the tab runner still submits transaction requests.
bazel run //tools/certification:runner -- setup or -- run — activates locations, imports credentials, creates sessions, and/or sends transaction traffic.
tools/certification/scripts/package-run.sh — uploads to GCS when gsutil is available.

Local end-to-end iteration loop

Use this flow when certification is close but a TSYS-required tag, saved-card path, or tab ordering bug still needs fast iteration. It runs the real services against the local emulators and the same certification runner code, then promotes only after the payloads are proven locally.

Which environment to use

Use the cheapest environment that can prove the question in front of you:

Question	Environment	Why
Does Kotlin build and do unit tests still pass?	Local Bazel	No network or payment side effects.
Does the runner produce the right tab/step sequence?	Local stack	Proves orchestration and run-state behavior.
Does checkout save a real database-backed payment method?	Local stack	Proves customer/token storage without spending staging transactions.
Does the TransIT request contain all required tags?	Local stack or staging dry-run/log export	The validator can catch missing tags before analyst review.
Does TSYS approve and return host reference numbers?	Staging against TransIT UAT	Requires the real TransIT endpoint.
Did settlement pass after batch close?	Staging against TransIT UAT	Requires the TSYS `:30` batch push.
Does Apple Pay show the native sheet on iPhone?	Deployed HTTPS checkout origin	Apple wallet requires a trusted web origin and device wallet state.

What local can prove

The cert runner builds the correct tab sequence, start step, stop step, and resume state.
Checkout/session flows create customers, save cards, and reuse stored payment methods through the database instead of temporary files.
Processing builds the TransIT request DTOs with required terminal, industry, brand, Level II, Level III, card-on-file, void, and tokenization fields.
The raw request body shape can be compared against TSYS requirements before sending another staging transaction.

What local cannot prove

Real TransIT approvals, declines, host reference numbers, settlement, or batch close status.
Cloud Run IAM, deployed Secret Manager wiring, VPC/networking, or revision selection.
Apple Pay device-sheet behavior. Local tests can validate our request shape, but a real iPhone wallet flow must hit a deployed HTTPS checkout origin that Apple trusts.

Start the local stack

# From the repo root.
./scripts/dev.sh services
./scripts/dev.sh status

Wait until auth, management, online-txn, processing, and the Spanner emulator are healthy. If a service fails during startup, tail only that service:

./scripts/dev.sh logs processing
./scripts/dev.sh logs online-txn

Prefer /opt/homebrew/bin/bazel on Apple Silicon developer laptops if the shell does not resolve the expected Bazel binary:

/opt/homebrew/bin/bazel build //tools/certification:runner

Create local cert credentials

Mint or reuse a local API token that has both public checkout scopes and admin setup scope. For most developer loops, use the same OAuth client configured by the local auth service and keep the tokens in shell variables only:

export CERT_PUBLIC_API_URL="http://localhost:8184"
export CERT_PUBLIC_MANAGEMENT_URL="http://localhost:8183"
export CERT_TRANSIT_MID=888000003647
export CERT_TRANSIT_TERMINAL_NUMBER=75959504
export CERT_DEVICE_ID=88800000364701
export CERT_ORGANIZATION_ID=<local-organization-id>
export CERT_LOCATION_ID=888000003647
export CERT_API_KEY=<local-hosted-payments-and-transactions-token>
export CERT_MANAGEMENT_API_KEY=<local-admin-token>

Those URLs match the ./scripts/dev.sh services host-port mappings: online-txn on 8184, management on 8183. The cert runner's built-in local defaults use the same ports when no URL flags or env vars are provided.

If the local database is empty, create the organization, merchant, location, and TransIT credential profile through the management API or admin portal first. Then run setup once:

/opt/homebrew/bin/bazel run //tools/certification:runner -- setup \
  --api-url "$CERT_PUBLIC_API_URL" \
  --management-url "$CERT_PUBLIC_MANAGEMENT_URL" \
  --api-key "$CERT_API_KEY" \
  --management-api-key "$CERT_MANAGEMENT_API_KEY" \
  --organization-id "$CERT_ORGANIZATION_ID" \
  --location-id "$CERT_LOCATION_ID" \
  --transit-mid "$CERT_TRANSIT_MID" \
  --transit-terminal-number "$CERT_TRANSIT_TERMINAL_NUMBER" \
  --device-id "$CERT_DEVICE_ID"

Do not put PAN, CVV, tokens, or raw TransIT payloads in shell history, chat, PR comments, or issue comments. Raw evidence belongs in the run folder or GCS only.

Run bounded slices

Use a run root outside tools/certification/cert-runs when iterating so the files are easy to keep and compare:

export RUN_ROOT="$HOME/Downloads/transit-local-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$RUN_ROOT"

Run the lowest-cost slice that exercises the bug:

# e-Commerce through the first manual checkpoint (the runner returns exit 3).
/opt/homebrew/bin/bazel run //tools/certification:runner -- run \
  --public-api-url "$CERT_PUBLIC_API_URL" \
  --public-management-url "$CERT_PUBLIC_MANAGEMENT_URL" \
  --api-key "$CERT_API_KEY" \
  --management-api-key "$CERT_MANAGEMENT_API_KEY" \
  --organization-id "$CERT_ORGANIZATION_ID" \
  --location-id "$CERT_LOCATION_ID" \
  --transit-mid "$CERT_TRANSIT_MID" \
  --transit-terminal-number "$CERT_TRANSIT_TERMINAL_NUMBER" \
  --device-id "$CERT_DEVICE_ID" \
  --tab ecommerce \
  --output-dir "$RUN_ROOT/ecommerce"

# After writing a signed schemaVersion=2 attestation, resume exactly ec11.
/opt/homebrew/bin/bazel run //tools/certification:runner -- run \
  --public-api-url "$CERT_PUBLIC_API_URL" \
  --public-management-url "$CERT_PUBLIC_MANAGEMENT_URL" \
  --api-key "$CERT_API_KEY" \
  --management-api-key "$CERT_MANAGEMENT_API_KEY" \
  --organization-id "$CERT_ORGANIZATION_ID" \
  --location-id "$CERT_LOCATION_ID" \
  --transit-mid "$CERT_TRANSIT_MID" \
  --transit-terminal-number "$CERT_TRANSIT_TERMINAL_NUMBER" \
  --device-id "$CERT_DEVICE_ID" \
  --tab ecommerce \
  --start-step ec11 \
  --operator-confirmation-file "$RUN_ROOT/ecommerce/attestation-ec11.json" \
  --output-dir "$RUN_ROOT/ecommerce"

# Wait for the TSYS :30 settlement window, then resume the business step.
/opt/homebrew/bin/bazel run //tools/certification:runner -- run \
  --public-api-url "$CERT_PUBLIC_API_URL" \
  --public-management-url "$CERT_PUBLIC_MANAGEMENT_URL" \
  --api-key "$CERT_API_KEY" \
  --management-api-key "$CERT_MANAGEMENT_API_KEY" \
  --organization-id "$CERT_ORGANIZATION_ID" \
  --location-id "$CERT_LOCATION_ID" \
  --transit-mid "$CERT_TRANSIT_MID" \
  --transit-terminal-number "$CERT_TRANSIT_TERMINAL_NUMBER" \
  --device-id "$CERT_DEVICE_ID" \
  --tab ecommerce \
  --start-step ec12_5 \
  --output-dir "$RUN_ROOT/ecommerce"

For broad regression checks, run all in-scope tabs from scratch. This is slower, but it catches tab-specific differences such as Direct Marketing PHONE, Direct Marketing MAIL, recurring, installment, and card-on-file request values:

for tab in ecommerce moto recurring; do
  /opt/homebrew/bin/bazel run //tools/certification:runner -- run \
    --api-url "$CERT_PUBLIC_API_URL" \
    --management-url "$CERT_PUBLIC_MANAGEMENT_URL" \
    --api-key "$CERT_API_KEY" \
    --management-api-key "$CERT_MANAGEMENT_API_KEY" \
    --organization-id "$CERT_ORGANIZATION_ID" \
    --location-id "$CERT_LOCATION_ID" \
    --transit-mid "$CERT_TRANSIT_MID" \
    --transit-terminal-number "$CERT_TRANSIT_TERMINAL_NUMBER" \
    --device-id "$CERT_DEVICE_ID" \
    --tab "$tab" \
    --output-dir "$RUN_ROOT/$tab"
done

Validate request payloads before staging

The local run output proves gateway behavior. For Gateway production-path evidence, use the self-contained production validator. It derives correlation IDs and the padded runner-artifact window, then performs its own authenticated Cloud Logging query; the raw export is only a parsing/compatibility input:

python3 tools/certification/scripts/validate-epx-required-fields.py \
  --raw-log "$RUN_ROOT/raw-gateway.json" \
  --run-dir "$RUN_ROOT/epx-production" \
  --expectations "$RUN_ROOT/epx-production/epx-expectations.json" \
  --output "$RUN_ROOT/epx-production/production-validation.txt"

The production validator ignores .filter, provenance, and requery sidecars. It internally queries /usr/bin/gcloud logging read against project central-log-monitor-ha444-hn61, bucket myriadventures-logging, location global, view _AllLogs, restricted to peakgateway-staging Gateway Cloud Run records and the runner correlation IDs.

The legacy TransIT validator below is diagnostic-only. It starts with UNTRUSTED DIAGNOSTIC — NOT CERTIFICATION EVIDENCE and exits 2 even when it finds zero errors; do not attach its output as certification evidence or call its clean result a pass.

Required-field validator workflow

The legacy validator is a fast feedback loop for the analyst's API feedback and Excel tab requirements, not a certification gate. Run it against raw TransIT request logs to diagnose DTO changes before asking TSYS to review a tab.

tools/certification/scripts/validate-transit-required-fields.py \
  --raw-log "$RUN_ROOT/raw-transit/ecommerce-transit-raw.json" \
  --run-dir "$RUN_ROOT/ecommerce" \
  --output "$RUN_ROOT/raw-transit/validation-ecommerce.txt"

tools/certification/scripts/validate-transit-required-fields.py \
  --raw-log "$RUN_ROOT/raw-transit/moto-transit-raw.json" \
  --run-dir "$RUN_ROOT/moto" \
  --output "$RUN_ROOT/raw-transit/validation-moto.txt"

tools/certification/scripts/validate-transit-required-fields.py \
  --raw-log "$RUN_ROOT/raw-transit/recurring-transit-raw.json" \
  --run-dir "$RUN_ROOT/recurring" \
  --output "$RUN_ROOT/raw-transit/validation-recurring.txt"

The diagnostic should report zero errors for a clean fixture, but its process status remains 2 and its banner remains untrusted. Warnings must be understood and written in the run notes. For example, missing serviceLocationGeoCoordinates is acceptable only when the merchant is not a traveling merchant; otherwise fix merchant or location metadata and rerun MasterCard transactions.

The validator should fail on production DTO problems, not cert-runner-only problems. If a required tag is missing, update one of these layers:

Missing field class	Fix location
Terminal/card-present tags	TransIT transaction profile resolver and request DTO mapping.
Brand fields such as MasterCard acceptor fields or Discover registration	Brand-specific profile mapping.
Level II or Level III fields	Commercial data DTO defaults and serializer ordering.
Card-on-file fields	Stored-token/customer flow and TransIT COF mapping.
Void fields	Void request DTO and processing operation mapping.

Promote to staging

After local payload validation passes:

Run focused Bazel tests for the touched service and Transit client.
Build and push the affected service image.
Deploy staging and confirm the expected Cloud Run revision is serving.
Re-run the same bounded tab slice against staging.
Export raw TransIT request/response logs from Cloud Logging into the run root.
Re-run the required-field validator against the exported raw logs.
Only then run the full three-tab certification pass and observe the TSYS :30 settlement wait before return/void steps.

Export staging raw logs

For staging evidence, export raw request/response logs from Cloud Logging into the run root. Keep these files private because TSYS requires the complete raw payloads and they may contain sensitive payment data.

export PROJECT_ID=peakgateway-staging  # staging project (cert runs target staging)
export RUN_ROOT="$HOME/Downloads/stage-<run-id>"
export START_TIME="<YYYY-MM-DDTHH:MM:SSZ>"
export END_TIME="<YYYY-MM-DDTHH:MM:SSZ>"

mkdir -p "$RUN_ROOT/raw-transit"

gcloud logging read \
  'resource.type="cloud_run_revision"
   resource.labels.service_name="gateway-processing"
   timestamp>="'"$START_TIME"'"
   timestamp<="'"$END_TIME"'"
   ("TransIT request" OR "TransIT response")' \
  --project "$PROJECT_ID" \
  --format=json \
  > "$RUN_ROOT/raw-transit/combined-transit-raw.json"

Then, for a local TransIT diagnostic only, run:

tools/certification/scripts/validate-transit-required-fields.py \
  --raw-log "$RUN_ROOT/raw-transit/combined-transit-raw.json" \
  --run-dir "$RUN_ROOT" \
  --output "$RUN_ROOT/raw-transit/validation-combined.txt"

This command starts with UNTRUSTED DIAGNOSTIC — NOT CERTIFICATION EVIDENCE and exits 2 even when it reports zero errors. For Gateway production-path evidence, use validate-epx-required-fields.py as shown above; it derives the runner correlation IDs/window and independently queries the pinned authenticated Cloud Logging view. Prefer tab-specific raw files for diagnostics. Do not paste raw log contents into GitHub, Slack, email body text, or chat; upload files through the TSYS Partner Portal or approved evidence bucket only when the certification owner explicitly requires them.

Real-run timestamp capture

When doing an analyst-submittable run, record these values as the run happens:

Value	Source
Tab name	Runner tab name, for example `eCommerce`, `MOTO`, or `Recurring`.
MID	Cert environment value, currently `888000003647`.
Authorization start	Timestamp of the first successful transaction request in the tab.
First batch close / pause	Runner output at the batch-close step.
Resume time	Timestamp when the post-`:30` resume run starts.
Authorization end	Timestamp of the final transaction before the tab is submitted.
First hostReferenceNumber	First TransIT response host reference number in the tab.
Last hostReferenceNumber	Last TransIT response host reference number in the tab.

Email format:

Hi <analyst>,

I completed the <tab name> tab and closed the batches manually. The first batch
was closed before the hourly :30 push window, then I waited until after the :30
settlement window before processing the return and return void steps.

Submission details:

Tab Name: <tab name>
MID: 888000003647
Authorization Start Date/Time: <Month D, YYYY H:MM:SS AM/PM timezone>
First Batch Close / Pause Time: <Month D, YYYY H:MM:SS AM/PM timezone>
Resume Time: <Month D, YYYY H:MM:SS AM/PM timezone>
Authorization End Date/Time: <Month D, YYYY H:MM:SS AM/PM timezone>
First hostReferenceNumber: <first hostReferenceNumber>
Last hostReferenceNumber: <last hostReferenceNumber>

I uploaded the complete raw TransIT request/response logs for this tab in the
Partner Portal.

Prerequisites

Staging location 888000003647 must be active. The /api/v1/locations/{id} endpoint lives on the public management surface, so confirm via the public management URL:
```
export CERT_PUBLIC_MANAGEMENT_URL="https://staging-api.peakgateway.co/management"

curl -sS -H "Authorization: Bearer $CERT_API_KEY" \
  -H "X-Merchant-ID: 888000003647" \
  "$CERT_PUBLIC_MANAGEMENT_URL/api/v1/locations/888000003647"
```
If not, create via the admin portal and run bazel run //tools/certification:runner -- setup ... once. Setup imports the Gateway credential profile; no provider-host pre-flight is required.
OAuth client for the runner. Credentials in Secret Manager:
- cert-runner-oauth-client-id
- cert-runner-oauth-client-secret
GCS bucket for evidence retention. gs://peak-cirunners-cert-runs/ with the GitHub Actions service account as roles/storage.objectCreator.

Full run (the happy path)

# 1. Export config
export CERT_PUBLIC_API_URL="https://staging-pay.peakgateway.co"
export CERT_PUBLIC_MANAGEMENT_URL="https://staging-api.peakgateway.co/management"

# 2. Fetch a fresh OAuth token
CLIENT_ID="$(gcloud secrets versions access latest --secret=cert-runner-oauth-client-id)"
CLIENT_SECRET="$(gcloud secrets versions access latest --secret=cert-runner-oauth-client-secret)"
export CERT_API_KEY="$(curl -sS -u "$CLIENT_ID:$CLIENT_SECRET" \
  -d grant_type=client_credentials \
  -d 'scope=hosted-payments:write transactions:write transactions:read' \
  "https://staging-api.peakgateway.co/oauth2/token" \
  | python3 -c 'import sys,json; print(json.load(sys.stdin)["access_token"])')"
export CERT_MGMT_API_KEY="$(curl -sS -u "$CLIENT_ID:$CLIENT_SECRET" \
  -d grant_type=client_credentials \
  -d 'scope=admin:*' \
  "https://staging-api.peakgateway.co/oauth2/token" \
  | python3 -c 'import sys,json; print(json.load(sys.stdin)["access_token"])')"
export CERT_ORGANIZATION_ID=<gateway-organization-id>
export CERT_LOCATION_ID=<gateway-location-id>
export CERT_TRANSIT_MID=888000003647
export CERT_TRANSIT_TERMINAL_NUMBER=75959504
export CERT_DEVICE_ID=88800000364701

# 3. Run the default automated tabs. This pauses with exit code 3 at the first manual checkpoint.
./tools/certification/run-all-tabs.sh

Expected: the script runs setup and the requested tab range, each writing to tools/certification/cert-runs/full-<timestamp>/<tab>/. When a tab reaches a manual batch-close checkpoint it prints the run root and exits 3; exit 3 is a paused/incomplete state, not a successful certification result. Use the exact persisted-checkpoint sequence in "Per-tab execution sequence" below, then generate the report/package only after all required attestations are confirmed.

Total wall clock: variable; settlement waits are operator-controlled. TransIT closes batches at 30 minutes past each hour.

TSYS review wants one tab per settlement. After each tab settlement, email the tab name, settlement date, and returned batch/settlement number to the TSYS reviewer.

Cloud Run Job per-tab real run

For a real certification pass, prefer one Cloud Run Job execution per tab with the evidence bucket mounted so each invocation appends to the same run root. The e-Commerce/MOTO/Recurring tabs have dependent business steps after their manual checkpoints. Do not resume e-Commerce directly at ec12_5 or MOTO directly at mo10_1 while ec11/mo9 is pending. First resume the exact checkpoint with its signed attestation, then wait for the TSYS :30 settlement window before resuming ec12_5/mo10_1; final ec14/mo13 and rc10 also require exact signed checkpoint resumes. If TSYS asks for an hour between tabs, wait outside Cloud Run between tab executions.

One-time bucket mount

export PROJECT_ID=<gcp-project-id>
export REGION=us-east1
export CERT_JOB=<cloud-run-cert-job-name>
export CERT_JOB_SA=<cert-job-service-account@${PROJECT_ID}.iam.gserviceaccount.com>
export CERT_RUN_BUCKET=peak-cirunners-cert-runs

gcloud storage buckets create "gs://${CERT_RUN_BUCKET}" \
  --project "$PROJECT_ID" \
  --location "$REGION" \
  --uniform-bucket-level-access

gcloud storage buckets add-iam-policy-binding "gs://${CERT_RUN_BUCKET}" \
  --member "serviceAccount:${CERT_JOB_SA}" \
  --role roles/storage.objectUser

gcloud run jobs update "$CERT_JOB" \
  --project "$PROJECT_ID" \
  --region "$REGION" \
  --add-volume name=cert-runs,type=cloud-storage,bucket="$CERT_RUN_BUCKET",mount-options="implicit-dirs=true" \
  --add-volume-mount volume=cert-runs,mount-path=/mnt/cert-runs

The entrypoint honors CERT_RUN_ROOT; set it to a path under the mounted bucket. If CERT_RUN_ROOT is not set, the job writes under /tmp, which is ephemeral and is not suitable for split-tab runs.

Per-tab execution sequence

Each job invocation below uses the same mounted CERT_RUN_ROOT. A runner pause is exit code 3, not a successful completion. The first invocation for a tab pauses at its manual checkpoint; resume the exact persisted checkpoint with a signed --operator-confirmation-file (passed through CERT_OPERATOR_CONFIRMATION_FILE by the Cloud Run entrypoint), wait for the TSYS settlement window when the runner pauses again, then resume the next business step. Never jump directly to a later business step while the checkpoint is PENDING.

export PROJECT_ID=<gcp-project-id>
export REGION=us-east1
export CERT_JOB=<cloud-run-cert-job-name>
export RUN_ID="full-$(date -u +%Y%m%dT%H%M%SZ)"
export CERT_RUN_ROOT="/mnt/cert-runs/${RUN_ID}"

# 1. Day 1 e-Commerce through ec11. This invocation must return exit 3.
gcloud run jobs execute "$CERT_JOB" \
  --project "$PROJECT_ID" \
  --region "$REGION" \
  --wait \
  --update-env-vars "CERT_RUN_ROOT=${CERT_RUN_ROOT},CERT_TABS=ecommerce,CERT_SKIP_SETUP=false,CERT_START_STEP=,CERT_OPERATOR_CONFIRMATION_FILE="

# 2. After writing a signed schemaVersion=2 attestation under the same tab directory,
#    resume exactly ec11. This invocation returns exit 3 for the TSYS settlement wait.
gcloud run jobs execute "$CERT_JOB" \
  --project "$PROJECT_ID" \
  --region "$REGION" \
  --wait \
  --update-env-vars "CERT_RUN_ROOT=${CERT_RUN_ROOT},CERT_TABS=ecommerce,CERT_SKIP_SETUP=true,CERT_START_STEP=ec11,CERT_OPERATOR_CONFIRMATION_FILE=${CERT_RUN_ROOT}/ecommerce/attestation-ec11.json"

# 3. After the TSYS :30 settlement window, resume e-Commerce business steps.
#    This reuses the same state and runs through ec14, where it pauses again.
gcloud run jobs execute "$CERT_JOB" \
  --project "$PROJECT_ID" \
  --region "$REGION" \
  --wait \
  --update-env-vars "CERT_RUN_ROOT=${CERT_RUN_ROOT},CERT_TABS=ecommerce,CERT_SKIP_SETUP=true,CERT_START_STEP=ec12_5,CERT_OPERATOR_CONFIRMATION_FILE="

# 4. Write attestation-ec14.json and resume the exact final checkpoint.
gcloud run jobs execute "$CERT_JOB" \
  --project "$PROJECT_ID" \
  --region "$REGION" \
  --wait \
  --update-env-vars "CERT_RUN_ROOT=${CERT_RUN_ROOT},CERT_TABS=ecommerce,CERT_SKIP_SETUP=true,CERT_START_STEP=ec14,CERT_OPERATOR_CONFIRMATION_FILE=${CERT_RUN_ROOT}/ecommerce/attestation-ec14.json"

# 5. Day 1 MOTO through mo9. This invocation must return exit 3.
gcloud run jobs execute "$CERT_JOB" \
  --project "$PROJECT_ID" \
  --region "$REGION" \
  --wait \
  --update-env-vars "CERT_RUN_ROOT=${CERT_RUN_ROOT},CERT_TABS=moto,CERT_SKIP_SETUP=true,CERT_START_STEP=,CERT_OPERATOR_CONFIRMATION_FILE="

# 6. Write attestation-mo9.json and resume exactly mo9; wait for the next TSYS :30.
gcloud run jobs execute "$CERT_JOB" \
  --project "$PROJECT_ID" \
  --region "$REGION" \
  --wait \
  --update-env-vars "CERT_RUN_ROOT=${CERT_RUN_ROOT},CERT_TABS=moto,CERT_SKIP_SETUP=true,CERT_START_STEP=mo9,CERT_OPERATOR_CONFIRMATION_FILE=${CERT_RUN_ROOT}/moto/attestation-mo9.json"

# 7. After settlement, resume MOTO business steps through mo13 (exit 3 at mo13).
gcloud run jobs execute "$CERT_JOB" \
  --project "$PROJECT_ID" \
  --region "$REGION" \
  --wait \
  --update-env-vars "CERT_RUN_ROOT=${CERT_RUN_ROOT},CERT_TABS=moto,CERT_SKIP_SETUP=true,CERT_START_STEP=mo10_1,CERT_OPERATOR_CONFIRMATION_FILE="

# 8. Write attestation-mo13.json and resume the exact final checkpoint.
gcloud run jobs execute "$CERT_JOB" \
  --project "$PROJECT_ID" \
  --region "$REGION" \
  --wait \
  --update-env-vars "CERT_RUN_ROOT=${CERT_RUN_ROOT},CERT_TABS=moto,CERT_SKIP_SETUP=true,CERT_START_STEP=mo13,CERT_OPERATOR_CONFIRMATION_FILE=${CERT_RUN_ROOT}/moto/attestation-mo13.json"

# 9. Recurring ends at rc10. It pauses with exit 3 until this signed exact-step resume.
gcloud run jobs execute "$CERT_JOB" \
  --project "$PROJECT_ID" \
  --region "$REGION" \
  --wait \
  --update-env-vars "CERT_RUN_ROOT=${CERT_RUN_ROOT},CERT_TABS=recurring,CERT_SKIP_SETUP=true,CERT_START_STEP=,CERT_OPERATOR_CONFIRMATION_FILE="

# 10. Write attestation-rc10.json and resume the exact final checkpoint.
gcloud run jobs execute "$CERT_JOB" \
  --project "$PROJECT_ID" \
  --region "$REGION" \
  --wait \
  --update-env-vars "CERT_RUN_ROOT=${CERT_RUN_ROOT},CERT_TABS=recurring,CERT_SKIP_SETUP=true,CERT_START_STEP=rc10,CERT_OPERATOR_CONFIRMATION_FILE=${CERT_RUN_ROOT}/recurring/attestation-rc10.json"

The signed files must contain the current runner fields (schemaVersion: 2, persisted run/tab/ step identity, route/profile/batch identity, signer key version/fingerprint, confirmation ID, evidence path/digest, and Ed25519 signature). The runner never signs on the operator's behalf, never auto-runs the next business step, and never calls a Gateway batch-close HTTP route.

Each invocation regenerates `cert-report.txt` from the state files already in
`CERT_RUN_ROOT`, so after the final tab the bucket should contain:

```text
gs://peak-cirunners-cert-runs/<RUN_ID>/
├── setup/
├── ecommerce/
├── moto/
├── recurring/
└── cert-report.txt

If a report needs to be regenerated without sending more transactions:

gcloud run jobs execute "$CERT_JOB" \
  --project "$PROJECT_ID" \
  --region "$REGION" \
  --wait \
  --update-env-vars "CERT_RUN_ROOT=${CERT_RUN_ROOT},CERT_REPORT_ONLY=true,CERT_SKIP_SETUP=true"

Partial runs

# Single tab
./tools/certification/run-all-tabs.sh --tabs moto

# Resume from a specific failed step
./tools/certification/run-all-tabs.sh --tabs ecommerce --start-step ec2_5

# Run through a specific step and pause
./tools/certification/run-all-tabs.sh --tabs ecommerce --stop-step ec11

# Skip setup (already activated)
./tools/certification/run-all-tabs.sh --skip-setup

# Run an explicit production-path slice without setup
./tools/certification/run-all-tabs.sh --skip-setup --tabs ecommerce

# Reuse an existing run root
./tools/certification/run-all-tabs.sh --run-root tools/certification/cert-runs/full-<timestamp> \
  --skip-setup --tabs ecommerce --start-step ec12_5

# Scope-dependent In-App tab, only when TSYS confirms it is in scope
./tools/certification/run-all-tabs.sh --skip-setup --tabs in-app

Triage: what to do when a step fails

The runner marks the step FAILED and stops the tab. Inspect the JSON log in the run directory:

cat tools/certification/cert-runs/full-<timestamp>/ecommerce/<step-id>.json

Common patterns and fixes:

Response pattern	Cause	Fix
`"responseStatus": 401`	Token expired	Re-fetch per "Full run" Step 2
`"responseStatus": 403` on webhook endpoints	Token missing scope	Add `webhook:manage` to the public API token scopes
`"responseStatus": 403` on `/management/api/v1/...`	Management token missing admin scope	Mint `CERT_MGMT_API_KEY` with `admin:*` and retry
`"responseStatus": 404` on `/api/v1/transactions/<id>/capture`	Transaction persistence is still in flight	Timing issue --- rerun the step with `--start-step <step-id>`
Status = `DECLINED` when `APPROVED` expected	Possible real TransIT decline --- check response code	If code is `A0000` the card was approved but our status mapping is wrong. Fix in `validateResponse` in `StepRunner.kt`. If code is anything else, escalate to Greg
Status = `APPROVED` when `PARTIAL_APPROVAL` expected	Partial-auth flag not set in request body	Confirm `partialAuthSupport: true` is in `buildSaleBody` for this step
`NullPointerException` in runner	Response shape mismatch --- a field the runner expected was null	Add null-safe extraction in the appropriate `execute*` method
Timeout on session creation	online-txn service cold start	Extend session creation timeout in `CertClient` from 60s to 120s. The staging `warm_services` setting keeps `min_instances >= 1` and fixes it permanently.

Submission to Global Payments

After a green full run (setup + e-Commerce, MOTO, and Recurring tabs):

./tools/certification/scripts/package-run.sh tools/certification/cert-runs/full-<timestamp>

This uploads the tarball to GCS and prints a signed URL (when CERT_RUN_SIGNURL_KEY points at a service-account JSON key). Email Greg Crosby (<Greg's email at Global Payments>) with:

Subject: Case 00230402 --- Pinpoint POS Cert Submission (run <timestamp>)
Body: a short summary, the signed URL, and a list of any steps that required notes.
Attachment: the filled-in cert Excel spreadsheet (populated by hand from cert-report.txt).

What this runbook covers​

EPX and EPG scope​

Red rows are skippable​

Local-only dry-run prep​

Local end-to-end iteration loop​

Which environment to use​

What local can prove​

What local cannot prove​

Start the local stack​

Create local cert credentials​

Run bounded slices​

Validate request payloads before staging​

Required-field validator workflow​

Promote to staging​

Export staging raw logs​

Real-run timestamp capture​

Prerequisites​

Full run (the happy path)​

Cloud Run Job per-tab real run​

One-time bucket mount​

Per-tab execution sequence​

Partial runs​

Triage: what to do when a step fails​

Submission to Global Payments​