Incident response

The job: something is wrong — a leaked secret, a runaway workload, a compromised account — and the response must be shorter than the time it takes to find someone with a browser. Each containment move below is one call, so it can live in a script in your repository and run from the incident channel, attributed to the person running it.

Prepare before the incident

Mint an incident-response token for each on-call engineer who is an Admin (Settings → API Tokens), with only the scopes the playbook needs:

Store it in the team's secret manager, and rehearse the scripts against a test project. Use a fixed, incident-scoped Idempotency-Key (for example incident-4711-…) so a panicked double-run is harmless.

The containment moves

Contain a leaked key

1. Find the key. If you have its masked prefix from a scanner alert, match it against the project's keys:

   Shell

   curl -fsS https://portal.<domain>/api/v1/mgmt/projects/$PROJECT_ID/keys \
     -H "Authorization: Bearer $TF_PAT" \
     | jq -r '.data[] | "\(.id)  \(.key_prefix)  \(.status)  \(.name)  last used \(.last_used_at)"'

   Copy

2. Delete it. confirmed_leak: true also cuts off requests already streaming on the key:

   Shell

   curl -fsS -X DELETE https://portal.<domain>/api/v1/mgmt/keys/$KEY_ID \
     -H "Authorization: Bearer $TF_PAT" \
     -H "Content-Type: application/json" \
     -H "Idempotency-Key: incident-$INCIDENT-delete-$KEY_ID" \
     -d '{"confirmed_leak": true}'

   Copy

3. Issue a replacement for the legitimate workload with POST /projects/{id}/keys.

Stop a runaway workload

curl -fsS -X POST https://portal.<domain>/api/v1/mgmt/projects/$PROJECT_ID/disable \
  -H "Authorization: Bearer $TF_PAT" \
  -H "Idempotency-Key: incident-$INCIDENT-disable-$PROJECT_ID"

import os
import requests

resp = requests.post(
    f"https://portal.<domain>/api/v1/mgmt/projects/{os.environ['PROJECT_ID']}/disable",
    headers={
        "Authorization": f"Bearer {os.environ['TF_PAT']}",
        "Idempotency-Key": f"incident-{os.environ['INCIDENT']}-disable-{os.environ['PROJECT_ID']}",
    },
    timeout=30,
)
resp.raise_for_status()
print(resp.json()["status"])  # "disabled"

When the cause is fixed, POST /projects/{id}/enable restores it. To cap rather than stop, lower guardrails.spend_limit_usd with PATCH /projects/{id}.

Contain a person

curl -fsS -X POST https://portal.<domain>/api/v1/mgmt/members/$MEMBER_ID/disable \
  -H "Authorization: Bearer $TF_PAT" \
  -H "Idempotency-Key: incident-$INCIDENT-disable-$MEMBER_ID"

API keys the person created keep working — they are project assets. If the person may have copied secrets, list the keys they created and rotate or delete them by id:

curl -fsS https://portal.<domain>/api/v1/mgmt/projects/$PROJECT_ID/keys \
  -H "Authorization: Bearer $TF_PAT" \
  | jq -r --arg uid "$MEMBER_ID" '.data[] | select(.created_by.id == $uid) | .id'

The last active admin cannot be disabled (409 last_admin), and you cannot disable yourself (409 cannot_act_on_self). To sign someone out of every browser without disabling them, use DELETE /members/{id}/sessions.

Afterwards: what happened

Pull the audit trail for the window, including every action taken through the API and which token took it:

curl -fsS -G https://portal.<domain>/api/v1/mgmt/audit/export \
  -H "Authorization: Bearer $TF_PAT" \
  -d format=jsonl \
  -d from=2026-09-24T08:00:00Z \
  -o incident-$INCIDENT-audit.jsonl

Every containment call above appears in it under the on-call engineer's name, with the token in metadata.pat_id / metadata.pat_name. An export covers at most 10,000 entries per call (422 export_too_large beyond that) — pass to as well and export in windows for a long incident. Changes to keys and projects take effect on the data plane within the 60-second propagation window.