Forms Portal — Incident Response Playbooks¶

Append to: mkdocs-portal/docs/response-plans/irp.md Classification: CONFIDENTIAL — Internal Use Only Version: 1.1 · 2026-08-31 Covers inventory IDs: gpus_forms_frontend, gpus_forms_backend, gpus_forms_db, gpus_forms_routing_worker, gpus_forms_approval_notify, gpus_forms_approval_notify_sub, gpus_forms_approval_notify_dlq, gpus_forms_approval_notify_dlq_sub

Each scenario below follows the established IRP format: Trigger → Severity → Initial response (first 15 min) → Investigation → Containment → Eradication → Recovery → Lessons learned.

FP-IR-01 — HappyFox credential leak¶

Trigger: HappyFox API key or auth code suspected exposed (git commit, shared screen, screenshot, phishing)

Severity: High — tickets can be spoofed; staff-submitted data exfiltratable

First 15 minutes: 1. Contact HappyFox support requesting immediate rotation (email support@happyfox.com + phone channel if available) 2. Disable the forms backend to stop further uses of the old credential: gcloud run services update gpus-forms-backend --region=us-central1 --no-traffic 3. Post #it-ops Slack channel with status 4. Check Cloud Logging for any unexpected HappyFox API calls in the last 24h

Investigation: - gcloud logging read 'resource.type=cloud_run_revision AND resource.labels.service_name=gpus-forms-backend AND textPayload:"happyfox"' --limit=500 --format=json - Cross-reference against expected submission volume from forms_submissions_total metric - If anomaly: list all HappyFox tickets created in the exposure window via HappyFox admin API

Containment: - HappyFox rotates — they provide new API key + auth code - Update Secret Manager: echo -n '<new-key>' | gcloud secrets versions add gpus-forms-happyfox-api-key --data-file=- - Repeat for auth code - Disable old secret versions: gcloud secrets versions disable <old-version> --secret=gpus-forms-happyfox-api-key

Eradication: - Force Cloud Run to pick up new secret: gcloud run services update gpus-forms-backend --region=us-central1 --update-env-vars=SECRET_REFRESH=$(date +%s) - Re-enable traffic: gcloud run services update gpus-forms-backend --region=us-central1 --traffic=latest=100

Recovery: - Submit a test form, verify ticket creates with new credentials - Audit all tickets from exposure window; close/duplicate any fraudulent ones via HappyFox admin

Lessons learned: - Review what caused exposure; update Secret Manager access policy if needed - Document in post-incidents/ with root cause + timeline

FP-IR-02 — Bulk submission export anomaly (Wazuh rule 100025)¶

Trigger: Wazuh alert 100025 fires (≥10 submission exports by a single actor within window)

Severity: High — potential insider data exfiltration

First 15 minutes: 1. SOC dashboard Tickets tab auto-creates ticket (per existing SOC rules, level≥10) 2. Identify actor: query audit_log filtered on action='submission_exported' for last 1h 3. Validate: is this a known admin doing legitimate export? Or unexpected? 4. If unexpected: revoke actor's forms portal access immediately

Investigation:

SELECT occurred_at, actor_username, actor_ip, details, request_id
FROM audit_log
WHERE action = 'submission_exported'
  AND occurred_at > NOW() - INTERVAL '24 hours'
ORDER BY occurred_at DESC;

- Cross-reference actor's recent auth_success / auth_failure events - Check if actor's Okta session came from unexpected IP or geo

Containment: - Revoke actor's Okta session: Okta Admin → Users → [actor] → Clear user sessions - Set users.is_active = false for the actor in forms DB - If compromise confirmed: force Okta password reset + MFA re-enrollment

Eradication: - Identify what was exported: details->>'submission_ids' in audit_log - Determine what data was in those submissions via decrypt (admin-only, audited) - Notify affected form owners per data classification policy

Recovery: - If data is recoverable (e.g., only staff names exposed internally): close with notification - If externally exposed: invoke DR-01 Data Breach playbook

Lessons learned: - Adjust Wazuh rule 100025 threshold if false-positive threshold was wrong - Consider rate-limiting /api/submissions export endpoints

FP-IR-03 — Decryption failure surge (Wazuh rule 100024)¶

Trigger: Wazuh alert 100024 fires repeatedly (multiple decrypt_failure events in audit_log)

Severity: Critical — possible KMS key rotation issue OR data tampering

First 15 minutes: 1. Check KMS key rotation state: gcloud kms keys versions list --key=gpus-forms-dek-wrapper --keyring=gpus-forms --location=us-central1 2. If a new key version was created recently (< 24h): rotation-related, benign-probable 3. If no recent rotation: possible tampering or DB corruption — escalate

Investigation: - SELECT occurred_at, target_id, details FROM audit_log WHERE action='decrypt_failure' ORDER BY occurred_at DESC LIMIT 100; - Which submission IDs fail? Isolated or widespread? - Query submission_fields: do kms_key_version values match current KMS state? - Check Cloud KMS audit logs: gcloud logging read 'resource.type=cloudkms_cryptokey AND protoPayload.methodName=~"Decrypt"'

Containment: - If key version mismatch: the old key version may have been auto-destroyed. Check gcloud kms keys versions list for DESTROYED state - If so: destroyed key version = affected submissions are unrecoverable. Identify scope. - If DB corruption: restore from latest backup to gpus-forms-db-recovery clone, compare submission_fields rows

Eradication: - If key issue: restore destroyed key version if still in 24h grace period (gcloud kms keys versions restore) - If corruption: restore affected rows from backup into production

Recovery: - Verify roundtrip decrypt on restored data: crypto.py roundtrip_test() endpoint - Notify submitters if their data needs re-submission (only if un-recoverable)

Lessons learned: - KMS rotation policy review - Consider per-submission DEK escrow for disaster scenarios (trade-off with security)

FP-IR-04 — Cloud SQL compromise / suspected unauthorized access¶

Trigger: Anomalous queries in Cloud SQL audit logs, unexpected schema changes, or Wazuh correlation alerts

Severity: Critical

First 15 minutes: 1. Disable forms backend traffic: gcloud run services update gpus-forms-backend --region=us-central1 --no-traffic 2. Capture current Cloud SQL state: gcloud sql backups create --instance=gpus-forms-db --description="IR-FP-IR-04-$(date +%s)" 3. Review Cloud SQL connection logs: gcloud logging read 'resource.type=cloudsql_database AND protoPayload.authenticationInfo.principalEmail!~"gpus-forms-backend"' 4. Check for unexpected IAM database users: gcloud sql users list --instance=gpus-forms-db

Investigation: - Compare current schema against expected (9 RLS policies, 12 tables, 4 roles): run the Q1-Q4 verification queries from Phase 1 Step 6 - Check audit_log for admin_reload_yaml or unexpected actions - Check if postgres built-in user has been password-set (signals someone elevated): SELECT rolname, rolvaliduntil FROM pg_roles WHERE rolname='postgres';

Containment: - Rotate postgres built-in user to throwaway random (same cleanup pattern from Phase 1 Step 6) - Revoke any unexpected IAM database users - Rotate the backend service account key (there isn't one today — we use ADC — but if keys were created, destroy them) - Re-verify RLS policies are intact; re-apply 002_rls.sql resume file if tampered

Eradication: - If data was altered: restore from pre-incident backup clone - Verify integrity by sampling decrypt roundtrip on submissions from the affected window

Recovery: - Re-enable traffic after all verifications pass - Post-mortem: how did attacker get in? Review VPC firewall rules, private IP exposure, service account key hygiene

Lessons learned: - Add Cloud SQL admin-activity alerts to Wazuh ruleset if not present

FP-IR-05 — KMS key compromise¶

Trigger: Cloud KMS key version showing unexpected decrypt calls, or compromise suspected

Severity: Critical — all submission data encrypted with that key version at risk

First 15 minutes: 1. Disable affected KMS key version: gcloud kms keys versions disable <version> --key=gpus-forms-dek-wrapper --keyring=gpus-forms --location=us-central1 2. Disable forms backend: gcloud run services update gpus-forms-backend --region=us-central1 --no-traffic 3. Capture KMS audit logs for the compromise window

Investigation: - gcloud logging read 'resource.type=cloudkms_cryptokey AND resource.labels.key_ring_id=gpus-forms' --freshness=7d - Identify all Decrypt calls + their principals - Cross-reference submission_fields rows with matching kms_key_version — these are the affected submissions

Containment: - Create new KMS key version: gcloud kms keys versions create --key=gpus-forms-dek-wrapper --keyring=gpus-forms --location=us-central1 - Update config.py KMS_KEY if key rotation behavior changed - Redeploy backend pointing at new version

Eradication: - Re-encrypt all affected submissions under new key version (batch job — requires admin-level process, NOT automatic) - Once all migrated, destroy compromised key version (30-day grace period)

Recovery: - Verify roundtrip on re-encrypted submissions - Re-enable traffic

Lessons learned: - Audit who had access to the key; review IAM bindings on gpus-forms-dek-wrapper - Consider HSM-backed keys if this recurs (increased cost)

FP-IR-06 — Forged or replayed approval decision¶

Trigger: A travel authorisation shows an approval nobody remembers making — an approver says they did not decide a step recorded against them, a status changed without a matching approval_decision_recorded audit row, or decided_by_email does not match the resolved_okta_email snapshot on the same row.

Severity: High — an unauthorised state transition on a live financial authorisation. approval_access.py exists because this is the threat; nothing above this line covered it.

Detection here is WEAK, and the gap is the finding

A denied attempt is recorded (audit_log.action = 'auth_failure', details.endpoint = 'approval_decision') and IS shipped to Wazuh — auth_failure is in soc_shipper_allowlist.json. But approval_access's reason vocabulary (no_matching_step, round_not_in_approval, no_token_email) is not in authz_reasons.json, so rules 100031–100036 do not match it. It lands on rule 100030 at level 3 — below the level ≥ 10 SOC auto-ticket threshold. Nobody is paged for a denied approval attempt.
A successful decision writes approval_decision_recorded, which is in models.py AUDIT_ACTIONS (33 values) but appears in neither ship_actions (17) nor excluded_actions (14) in soc_shipper_allowlist.json — 31 of 33 declared. approval_decision_recorded and approval_notification_sent are undeclared, so a recorded approval decision never reaches the SOC stream at all. The allowlist's own header says "a new enum value forces a ship decision"; for these two that decision was never taken.
ASVS fact G: forms_app holds UPDATE on approval_steps and there is no REVOKE anywhere in forms-backend/schema/. A decision can be rewritten, and the row carries no evidence that it was.

So the realistic detection is a human noticing — an approver disputing a decision attributed to them, or Kevin querying at step 3. That is the honest statement. Do not read the queries below as monitors; they are forensics you run after someone tells you. Raising the two shipping gaps is PRG-030 / PRG-031 work, not an IR step.

First 15 minutes: 1. Do not modify the row. approval_steps is mutable and carries no history; an investigative UPDATE destroys the only copy of the disputed state. 2. Snapshot the round before anything else:

SELECT * FROM approval_steps  WHERE submission_id = :id ORDER BY version, step_index;
SELECT * FROM submission_approvals WHERE submission_id = :id ORDER BY version;

Save to a file with a timestamp. This is the evidence. 3. If the round is still IN_APPROVAL, tell the coordinator (Shereyll) to hold — do not let step 2 or step 3 act on a disputed step 1. 4. If the trip is already booked, escalate to Kevin the same hour. Booking against a forged authorisation is the loss, not the database row.

Investigation:

-- 1. The immutable half. Does a decision row have a matching audit row?
SELECT s.step_index, s.status, s.decided_by_email, s.decided_at,
       s.resolved_okta_email, a.id AS audit_id, a.actor_username, a.actor_ip,
       a.details->>'action_taken' AS audited_action, a.occurred_at
  FROM approval_steps s
  LEFT JOIN audit_log a
         ON a.action = 'approval_decision_recorded'
        AND a.target_id = s.submission_id::text
        AND (a.details->>'step_index')::int = s.step_index
 WHERE s.submission_id = :id
 ORDER BY s.version, s.step_index;

* audit_id NULL on a decided step — the decision was written outside the endpoint. The endpoint writes both in one transaction (approval_decide.py:494-499), so this combination cannot arise from the application path. Treat as confirmed tampering. * decided_by_email ≠ resolved_okta_email — the four conditions require equality (condition 3), so the endpoint cannot produce this either. * audited_action ≠ the recorded status — the row was rewritten after the audit row was fixed.

-- 2. Denied attempts against this submission, at any level.
SELECT occurred_at, actor_username, actor_ip, details->>'reason' AS reason
  FROM audit_log
 WHERE action = 'auth_failure'
   AND details->>'endpoint' = 'approval_decision'
   AND target_id = :id
 ORDER BY occurred_at;

actor_ip may be the Cloud Run front-end rather than the caller — the proxy bug that blocks rules 100037/100038 also degrades this column. Corroborate from Okta, not here.

# 3. The replay path, which is BENIGN and must be excluded first.
gcloud logging read 'resource.labels.service_name="gpus-forms-backend"
  AND textPayload:"approval_decision.replay"' --freshness=30d

A 409 already_decided is the correct handling of a double-submitted decision by the same approver. Rule it out before treating a duplicate as an attack.

Containment: - Revoke the suspected actor's Okta session (Okta Admin → Users → [actor] → Clear user sessions) and set users.is_active = false in the forms DB. - If the identity is forms_app itself rather than a person — i.e. the write came through the backend with no audit row — the exposure is the Cloud Run service account, not a user. Rotate nothing yet; capture the revision first: gcloud run revisions list --service=gpus-forms-backend --region=us-central1 - Do not "correct" the step by editing it. If the round must not proceed, block it at the round: set submission_approvals.state = 'BLOCKED' with a blocked_reason naming this incident. That is a visible stop; a rewritten step is another undetectable write.

Eradication: - Establish the true decision from the approver directly, in writing, out of band. - Record the correction as a new audit row and a fresh round version — never by amending version 1. version is 1 everywhere today, so this is the first real use of the column; it is the right one. - If the entry point was a compromised Okta account, continue into FP-IR-07.

Recovery: - Re-run the round from the step below the disputed one. - Confirm the notification columns advanced (§ How you know it's working, check 2). - If a trip was booked on the forged approval, that is a finance matter — hand to Kevin with the snapshot from step 2 attached.

Lessons learned: - Ship approval_decision_recorded to Wazuh. Until it is in ship_actions, every successful approval decision in this estate is invisible to the SOC. Regenerate via soc/forms-authz-detection/gen_authz_detection.py; do not hand-edit the allowlist. - Add approval_access's reasons to authz_reasons.json so a denied approval attempt tiers above level 3. - Re-open the ASVS fact G question: which approval_steps columns are append-only. notification_sent_at and administrative re-dispatch are the legitimate post-decision writes; status, decided_by_email, decided_at and comment are not.

FP-IR-07 — Approver account compromise (Okta)¶

Trigger: Okta reports suspicious authentication for an account in approval_role_members — impossible travel, MFA fatigue, a phishing report, or an approver saying they did not sign in.

Severity: High — for the approval workflow, Okta is the whole control.

Why this is a separate playbook from FP-IR-02

FP-IR-02 revokes an Okta session for a data-export actor. It does not cover an approver, and irp.md describing FP-IR-01–05 as including "Okta token compromise" was wrong — grep -ic approval forms-portal-ir.md returned 0 before this section existed.

The reason this needs its own path is ASVS fact C: the approval link (https://forms.greenpeace.us/approvals/<submission_id>) carries no token, no query parameter and no secret. It is not a bearer credential and cannot be revoked — there is nothing to revoke. Authorisation is recomputed server-side on every call against the caller's Okta-asserted identity. Therefore: killing the Okta session is not one containment option among several. It is the only one. There is no link to expire, no token to rotate, no per-request secret to burn.

First 15 minutes: 1. Okta Admin → Users → [approver] → Clear user sessions, then suspend the account. Until this is done the attacker can decide any step dispatched to that identity. 2. Enumerate what is currently exposed to that identity — the steps they could decide right now:

SELECT s.submission_id, s.version, s.step_index, s.status, s.dispatched_at
  FROM approval_steps s
  JOIN submission_approvals r
    ON r.submission_id = s.submission_id AND r.version = s.version
 WHERE lower(s.resolved_okta_email) = lower(:approver_email)
   AND s.status = 'DISPATCHED'
   AND r.state  = 'IN_APPROVAL';

All four conditions must hold for a decision (approval_access § the four conditions); DISPATCHED + IN_APPROVAL is the pair that makes this list live. 3. Check whether any of them were decided during the compromise window — run FP-IR-06's query 1 against each submission_id returned.

Investigation: - Okta System Log is the primary source, not the forms portal. The portal records token_email from an already-validated JWT; it cannot tell a legitimate sign-in from a compromised one. - sql SELECT occurred_at, action, target_id, details->>'action_taken' AS act, actor_ip FROM audit_log WHERE actor_username = :username AND occurred_at >= :window_start ORDER BY occurred_at; Covers reads (submission_viewed) and — once the shipping gap in FP-IR-06 is closed — decisions. Today approval_decision_recorded is in the table but not in the SOC stream, so this query is the only place it surfaces. - Snapshot authorization does not revoke (ASVS fact D). Removing the person from approval_role_members does not withdraw their access to steps already dispatched — resolved_okta_email was snapshotted at dispatch. Suspending the Okta account is what closes it; the roster edit is bookkeeping.

Containment: - Okta: suspend, clear sessions, force password reset + MFA re-enrolment. - For every step from query 2 that must not proceed: block the round (submission_approvals.state = 'BLOCKED', blocked_reason naming the incident). Do not edit the step. - Do not set FORMS_APPROVAL_RECIPIENT_OVERRIDE to "protect" the queue. It is global — every approval mail estate-wide would divert, and unrelated approvers would silently stop hearing about real requests.

Eradication: - Restore the account only after Okta confirms the initial access vector is closed. - Re-dispatch each blocked round; re-dispatch is how revocation happens at all (fact D). - If any decision was made by the attacker, run FP-IR-06 for that submission.

Recovery: - Confirm the approver can reach /approvals/<id> and that the decision buttons render (the SPA reads can_decide, which the handler recomputes server-side). - Verify the re-dispatched steps show notification_published_at and notification_sent_at set.

Lessons learned: - The roster is the blast radius. approval_role_members should be reviewed on the same cadence as any privileged group; today nothing reviews it. - Fact D deserves an operational control: a documented "revoke an approver" procedure that names Okta suspension as step 1 and the roster edit as step 2, in that order.

FP-IR-08 — Notification pipeline stall (published, never sent)¶

Trigger: approval_steps.notification_published_at is SET while notification_sent_at is NULL, on a step older than a few minutes. Equivalently: an approver says they were never told about a request the portal shows as DISPATCHED.

Severity: High — silent. The rows read as correct, the SOC is quiet, and the request simply never moves. This has happened at least three times:

Date	What stalled	Mechanism
2026-08-18 16:33:43	submission `bb575538`	`dispatch_step` sent mail from Cloud Run, whose Postfix reach is nil — `[Errno 111] Connection refused`. Fixed by `1887fa0`, which made the backend a publisher and moved the send to MAPLE rather than opening Postfix to the VPC
2026-08-18	every published pointer	Both approval drains were unreachable (`194a9e2`). `drain_approval_subscription` was the last statement of the worker loop, after three `continue`s — including `DeadlineExceeded`, the normal idle path at `PULL_TIMEOUT_S=30`. `drain_approval_dlq` sat behind a bare `return` in an exception branch. Both had executed zero times. The boot log reported the subscription as configured throughout
2026-08-19	rounds `30a286e1`, `6418f160`	Both went terminal (APPROVED / RETURNED_FOR_REVISION, `tgarg@greenpeace.org`) before migration 019 added `outcome_notified_at`. Same class one level up: two travellers were never told the outcome. The deploy does not reach back; they still need a manual backfill

The 2026-08-18 defect is why 'no alerts' is not evidence

It survived verification because the drains logged nothing on an empty pull — "not called", "called and idle" and "called and timed out" were indistinguishable, and the absence of approval.drain_failed was read as proof the IAM binding worked. It proved nothing. The heartbeat exists so that silence is now falsifiable.

The four Pub/Sub assets carry monitoring_status: planned. There is no DLQ-depth alert. A message stranded in gpus-forms-approval-notify-dlq today is visible only to someone who goes and looks. Detection for this scenario is a person running the query below — usually because R2 flagged the request as stalled to Shereyll, or an approver asked. That is the gap; it is not papered over here.

First 15 minutes: 1. Is the worker alive at all?

journalctl -u gpus-forms-routing-worker -n 200 | grep 'approval.drain.heartbeat'

Expected every DRAIN_HEARTBEAT_S (300s), reporting iterations / messages / errors since boot, and immediately on the first record after a restart. Absent ⇒ the drain is not running. Not "running and idle" — those were made distinguishable precisely because confusing them is how the 2026-08-18 defect shipped. Expect approval.dlq_drain.heartbeat alongside it. 2. Scope it — one step, or everything?

SELECT submission_id, version, step_index, status,
       notification_published_at, notification_sent_at
  FROM approval_steps
 WHERE notification_published_at IS NOT NULL
   AND notification_sent_at IS NULL
 ORDER BY notification_published_at;

3. Check the override, because it produces rows that look correct:

journalctl -u gpus-forms-routing-worker | grep 'boot approval_recipient_override'

Expected: boot approval_recipient_override=OFF — LIVE DELIVERY to approvers. If it reads ACTIVE to=…, no approver is receiving anything — steps still reach DISPATCHED and notification_sent_at is still stamped. Absent entirely ⇒ the worker did not reach its boot banner or is running an older build, and unknown is not OFF.

Investigation:

Read the column pair as the platform doc's diagnostic table does — it distinguishes four states and only one of them is a mail problem:

published	sent	Meaning	Where to look
NULL	NULL	Never published; the dispatch did not complete or the publish failed	Cloud Run backend logs. Row is `DISPATCHED` and recoverable — this is the state that says so
SET	NULL	This scenario. Published; the worker has not sent	Heartbeat, then the DLQ subscription
SET	SET	Handed to the relay. Not delivery (ASVS fact E) — bounces downstream are invisible to the portal	`/var/log/maillog` on MAPLE
NULL	SET	Should not occur	Data-integrity question, not a mail question

# Is it stranded in the DLQ, or still retrying?
gcloud pubsub subscriptions pull gpus-forms-approval-notify-dlq-sub \
  --limit=10 --format=json          # DO NOT --auto-ack while investigating

The payload is a pointer — {submission_id, version, step_index} only. No rendered body, no address, no amount. The message tells you which step stalled and nothing about its content; that is deliberate (Pub/Sub messages are durable and retained).

-- Did it actually go out, and to whom?
SELECT created_at, target_id, details->>'intended', details->>'delivered_to',
       details->>'redirected'
  FROM audit_log
 WHERE action = 'approval_notification_sent' AND target_id = :id
 ORDER BY created_at DESC;

No approval_notification_sent row for a step whose notification_sent_at IS stamped ⇒ the two facts diverged, which the single-transaction write is supposed to make impossible. Investigate the worker, not the mail.

Containment: - If the worker is down: systemctl restart gpus-forms-routing-worker, then confirm the heartbeat within one iteration and the override=OFF banner. Restart alone is not the fix if the cause was reachability — read the loop, not the log level. - If the override is ACTIVE and should not be: clear it in /etc/gpus-forms-routing.env. The backend's override is read per send; the worker's is a module constant and needs the restart. Confirm the banner afterwards. - While stalled, tell the affected approvers directly. The request is real and pending; they are the only ones who can move it.

Eradication: - Root-cause into one of: reachability (statement placement — check that the drains still sit ahead of the routing pull and at duty 0 of sweep(); the comments there say ORDER IS LOAD BEARING), IAM on the subscription, or SMTP on MAPLE. - Re-drive the DLQ once the cause is fixed. The send is guarded AND notification_sent_at IS NULL, so a redelivery cannot double-send a step already sent.

Recovery: - Both columns set, published before sent, for every affected step. - Terminal rounds with outcome_notified_at still NULL are the second recovery query — every traveller whose request is finished and who was never told:

SELECT submission_id, version, state, outcome_notified_at
  FROM submission_approvals
 WHERE state IN ('APPROVED','RETURNED_FOR_REVISION','DECLINED')
   AND outcome_notified_at IS NULL;

30a286e1 and 6418f160 are expected here until backfilled.

Lessons learned: - Promote the four Pub/Sub assets off monitoring_status: planned. A DLQ-depth alert is the difference between this scenario having a detection signal and not having one. - The heartbeat is the model: state the normal case out loud at a readable cadence, so silence becomes evidence. Anything else added to this pipeline should do the same. - Backfill 30a286e1 and 6418f160.

FP-IR-09 — BLOCKED round accumulation¶

Trigger: submission_approvals.state = 'BLOCKED' on one or more rounds. In practice: a traveller asks why nothing has happened, or someone runs the query below.

Severity: Medium — no data is exposed and nothing is wrong with the estate's security posture. It is a availability and integrity-of-process failure: the request stops, and there is no code path out of BLOCKED.

There is no detection, and no exit. Both are the finding.

No detection. BLOCKED writes no audit_log row, no Wazuh rule matches it, and no report counts it. R2 is a chase-list of stalled requests, R5 counts volume and cycle time, R1 groups submissions by bounded keys — a BLOCKED round is in none of their numerators as a distinct state. The only way anyone learns is that a person asks. Writing "monitor for BLOCKED" here would be inventing a monitor; instead, run the query below on a schedule you actually keep.
No exit. There is no unblock endpoint. WITHDRAWN has been a legal state since migration 014 and no code path has ever set it — migration 021 set it by hand on four test chains. So a BLOCKED round is stranded until someone edits the database.

Seven conditions produce BLOCKED (approval_resolver.resolve_chain_core; 5 and 6 are re-evaluated at dispatch, where the same failure blocks the round and publishes nothing):

#	Condition	Most likely cause
1	`form_id` has no chain configured	A new form with an approval expectation and no chain
2	Submission row not found	Data-integrity problem — investigate as one
3	No `SMTApprover` value in `searchable_values`	Field removed, renamed, or submitted blank
4	Display value has no row in `approval_role_display_map`	The common one. A rename in `forms/pulldowns.yaml` not mirrored in the map
5	A role has zero active members	Departure, or `is_active` cleared
6	A role has more than one active member	Two people seeded for one role; ambiguous
7	Every step resolved to a skip	No approver remains after `SKIPPED_SELF` / `SKIPPED_DUPLICATE`

The join is exact string equality on a human display name — no strip(), no casefold(), no normalisation. That is deliberate: quietly repairing a near-miss would hide the drift BLOCKED exists to surface. Felicity Vonsuck is one word; an earlier draft carried Felicity Von Suck, which would have resolved to nothing on every submission naming her.

First 15 minutes: 1. Find them all, and read blocked_reason — it records the unmatched string verbatim, and for condition 6 it names the competing addresses:

SELECT submission_id, version, state, blocked_reason, created_at
  FROM submission_approvals
 WHERE state = 'BLOCKED'
 ORDER BY created_at;

2. Age them. A round blocked for weeks means a traveller has been waiting for weeks and nobody knew — that is the incident, more than the block itself. 3. Tell each submitter their request has not moved. They are the only party currently receiving no signal at all: no approver was mailed, so no outcome mail exists either.

Investigation:

For condition 4 — compare the two vocabularies directly:

SELECT DISTINCT r.blocked_reason
  FROM submission_approvals r
 WHERE r.state = 'BLOCKED';

SELECT display_value, role_key FROM approval_role_display_map ORDER BY display_value;

python3 -c "import yaml;d=yaml.safe_load(open('forms/pulldowns.yaml'));\
print([p['values'] for p in d if p['name']=='Travel SMT Approver'])"

The six seeded values were verified byte-exact against the repo YAML on 2026-08-18 — all ASCII, no leading or trailing whitespace, lengths 12/12/16/12/14/12. A mismatch now means one side moved since; find which, and fix that side rather than loosening the comparison.

For conditions 5 and 6:

SELECT role_key, count(*) FILTER (WHERE is_active) AS active,
       string_agg(okta_email, ', ') FILTER (WHERE is_active) AS members
  FROM approval_role_members GROUP BY role_key ORDER BY role_key;

Exactly one active member per role is the only healthy state. travel_admin_ops and travel_final are fixed roles resolved from this table directly and never pass through the display map — a problem there is always 5 or 6, never 4.

Containment: - There is nothing to contain: a blocked round has published nothing and mailed nobody. The correct action is to stop new rounds hitting the same condition — fix the map row or the roster before clearing the backlog, or you will re-block them. - Never default a blocked round to a fallback approver and never drop the step silently. The visible stop is the design.

Eradication: - Fix the cause: add the missing approval_role_display_map row (condition 4), or correct approval_role_members to exactly one active member (5, 6), or configure the chain (1). - Then re-resolve the affected rounds. There is no endpoint for this — it is a manual re-run of resolve_chain_core plus a state write, performed as maple-agent, one round at a time, with the before-state captured first. - If a blocked request is stale enough to be moot, WITHDRAWN is the honest terminal state — but note it must be set by hand, it emits no submitter notification (_SENDABLE_OUTCOMES has no WITHDRAWN and the renderer has no branch for it), and it records neither actor nor reason. Tell the traveller yourself. Do not delete the row.

Recovery: - Round back to IN_APPROVAL, step 1 DISPATCHED, both notification columns set. - Re-check the BLOCKED query returns nothing unexpected.

Lessons learned: - The BLOCKED query belongs on a schedule. Until something counts blocked rounds, the detection for this scenario is "a traveller complains", and that is what this section says rather than pretending otherwise. - A rename in forms/pulldowns.yaml touching Travel SMT Approver must be mirrored in approval_role_display_map in the same change. Consider a coverage-check-style guard comparing the two, in the spirit of scripts/check-component-coverage.py. - The undesigned withdraw endpoint (four open design questions, per platform/travel-approval-workflow.md § What is NOT built) is what would give this scenario a supported exit. When built it must add an approval_withdrawn audit action.

References¶

Existing IRP: mkdocs-portal/docs/response-plans/irp.md
Approval workflow, state machine and failure modes: platform/travel-approval-workflow.md
ASVS L2 scope statement, declared facts A–G: security/asvs-scope-forms-approval.md
Forms authz detection (Wazuh rules 100030–100039, shipper allowlist): soc/forms-authz-detection/
Direct DB access as maple-agent: infrastructure/runbooks/forms-db-access.md
Tabletop playbooks: mkdocs-portal/docs/response-plans/tabletop-playbooks.md
SOC auto-ticketing thresholds: SOC dashboard Tickets tab
Cloud SQL IAM DB user recovery (if postgres password lost): recreate via gcloud sql users set-password postgres --instance=gpus-forms-db --password=<new> with project owner credentials