Skip to main content

Reading cert-runner Cloud Run Job logs

Audience: Anyone monitoring a gateway-cert-runner-staging run who opened the Cloud Run console's Logs tab and found it empty.

Tool: gcloud logging read (or the Logs Explorer URL below).

Why the console Logs tab is empty: the gateway-cert-runner-staging Cloud Run Job — and every other Cloud Run service in pinpoint-gateway — sends container stdout/stderr to a different GCP project for centralized logging. The console's built-in Logs tab queries the local project's _Default log bucket, which is empty for these services because the project-level _Default sink redirects everything out.

Where the logs actually live​

FieldValue
Projectcentral-log-monitor-ha444-hn61
Bucketmyriadventures-logging
Locationglobal
View_AllLogs

This is configured by the org-level _Default log sink:

$ gcloud logging sinks describe _Default --project=pinpoint-gateway
destination: logging.googleapis.com/projects/central-log-monitor-ha444-hn61/locations/global/buckets/myriadventures-logging
filter: NOT LOG_ID("cloudaudit.googleapis.com/...") AND NOT LOG_ID("externalaudit.googleapis.com/...")

So all non-audit logs (= all container output) leave pinpoint-gateway as soon as they are written.

Read-access prerequisite (one-time per identity)​

The central project does not auto-grant pinpoint-gateway users any view. Before any read works, an owner of central-log-monitor-ha444-hn61 (e.g. pius@myriadventures.group) must grant two bindings to the identity that wants to read:

TARGET=user:<email>            # the identity that wants log access

gcloud projects add-iam-policy-binding central-log-monitor-ha444-hn61 \
  --member="$TARGET" \
  --role=roles/logging.viewer \
  --condition=None

gcloud logging views add-iam-policy-binding _AllLogs \
  --bucket=myriadventures-logging \
  --location=global \
  --project=central-log-monitor-ha444-hn61 \
  --member="$TARGET" \
  --role=roles/logging.viewAccessor

Both are required. Project-level roles/logging.viewer alone returns PERMISSION_DENIED: Permission denied for all log views because the bucket has restricted views — you also need roles/logging.viewAccessor on the specific view. Granting only the view accessor also fails. (Verified live: accounts@ascade.co was granted both on 2026-05-08.)

How to read​

Latest execution​

EXEC=$(gcloud run jobs executions list \
  --job=gateway-cert-runner-staging \
  --project=pinpoint-gateway --region=us-east1 \
  --limit=1 --format='value(name)')

gcloud logging read \
  "labels.\"run.googleapis.com/execution_name\"=\"$EXEC\"" \
  --project=central-log-monitor-ha444-hn61 \
  --bucket=myriadventures-logging \
  --location=global --view=_AllLogs \
  --order=asc --limit=200 \
  --format='value(timestamp,textPayload)'

Settlement events (full cert run)​

gcloud logging read \
  "resource.labels.job_name=\"gateway-cert-runner-staging\" AND textPayload:\"settled\"" \
  --project=central-log-monitor-ha444-hn61 \
  --bucket=myriadventures-logging \
  --location=global --view=_AllLogs \
  --freshness=4h --order=asc \
  --format='value(timestamp,textPayload)'

The runner prints Batch <id> settled. at each successful settlement (StepRunner.kt:645). Five are expected per default 3-tab run: ec11, ec14, mo9, mo13, rc10 (after PRs #777, #778).

Errors only​

gcloud logging read \
  "resource.labels.job_name=\"gateway-cert-runner-staging\" AND severity>=ERROR" \
  --project=central-log-monitor-ha444-hn61 \
  --bucket=myriadventures-logging \
  --location=global --view=_AllLogs \
  --freshness=24h --order=desc --limit=20 \
  --format='value(timestamp,severity,textPayload)'

Service-side errors during a cert step​

When the cert runner reports a non-200 (e.g. a 500 from management, or a TransIT-side error), the matching server log is in the same bucket — query by service:

gcloud logging read \
  "resource.type=cloud_run_revision AND resource.labels.service_name=\"gateway-merchant-onboarding-staging\" AND severity>=WARNING AND timestamp>=\"<run-start>\"" \
  --project=central-log-monitor-ha444-hn61 \
  --bucket=myriadventures-logging \
  --location=global --view=_AllLogs \
  --order=asc --format=json

Substitute gateway-management-staging, gateway-online-txn-staging, gateway-processing-staging, etc. depending on which surface the cert runner hit.

Console UI​

Open this URL — it sets the project, bucket, view, and a query scoped to cert-runner logs:

https://console.cloud.google.com/logs/query;query=resource.labels.job_name%3D%22gateway-cert-runner-staging%22;storageScope=storage,projects%2Fcentral-log-monitor-ha444-hn61%2Flocations%2Fglobal%2Fbuckets%2Fmyriadventures-logging%2Fviews%2F_AllLogs?project=central-log-monitor-ha444-hn61

If the project picker complains, switch to central-log-monitor-ha444-hn61 manually first. The Refine scope dropdown should already be set to the myriadventures-logging bucket / _AllLogs view via the URL params.

Why gcloud logging tail does not work here​

gcloud logging tail is a separate gcloud component (gcloud components install logging) and is not available on most boxes. Either install it, or fall back to a polling loop on gcloud logging read --freshness=1m every 30s.

Suggested follow-ups (out of scope here)​

  • Add a structured-logging filter to the cert runner so settlement events emit as jsonPayload with explicit batchNumber / tabName fields, removing the textPayload:"settled" substring match.
  • Add a Cloud Logging log-based alert that pings Slack/email on Batch * settled. so the operator does not have to babysit the multi-hour run.
  • Either grant pinpoint-gateway developers a default group binding on roles/logging.viewer + roles/logging.viewAccessor in the central project, or route cert-runner logs to a pinpoint-gateway-local bucket via an additional sink.

Related​