Forms Portal — Incident Response Playbooks¶

Append to: mkdocs-portal/docs/response-plans/irp.md Classification: CONFIDENTIAL — Internal Use Only Version: 1.0 · 2026-04-18

Each scenario below follows the established IRP format: Trigger → Severity → Initial response (first 15 min) → Investigation → Containment → Eradication → Recovery → Lessons learned.

FP-IR-01 — HappyFox credential leak¶

Trigger: HappyFox API key or auth code suspected exposed (git commit, shared screen, screenshot, phishing)

Severity: High — tickets can be spoofed; staff-submitted data exfiltratable

First 15 minutes: 1. Contact HappyFox support requesting immediate rotation (email support@happyfox.com + phone channel if available) 2. Disable the forms backend to stop further uses of the old credential: gcloud run services update gpus-forms-backend --region=us-central1 --no-traffic 3. Post #it-ops Slack channel with status 4. Check Cloud Logging for any unexpected HappyFox API calls in the last 24h

Investigation: - gcloud logging read 'resource.type=cloud_run_revision AND resource.labels.service_name=gpus-forms-backend AND textPayload:"happyfox"' --limit=500 --format=json - Cross-reference against expected submission volume from forms_submissions_total metric - If anomaly: list all HappyFox tickets created in the exposure window via HappyFox admin API

Containment: - HappyFox rotates — they provide new API key + auth code - Update Secret Manager: echo -n '<new-key>' | gcloud secrets versions add gpus-forms-happyfox-api-key --data-file=- - Repeat for auth code - Disable old secret versions: gcloud secrets versions disable <old-version> --secret=gpus-forms-happyfox-api-key

Eradication: - Force Cloud Run to pick up new secret: gcloud run services update gpus-forms-backend --region=us-central1 --update-env-vars=SECRET_REFRESH=$(date +%s) - Re-enable traffic: gcloud run services update gpus-forms-backend --region=us-central1 --traffic=latest=100

Recovery: - Submit a test form, verify ticket creates with new credentials - Audit all tickets from exposure window; close/duplicate any fraudulent ones via HappyFox admin

Lessons learned: - Review what caused exposure; update Secret Manager access policy if needed - Document in post-incidents/ with root cause + timeline

FP-IR-02 — Bulk submission export anomaly (Wazuh rule 100025)¶

Trigger: Wazuh alert 100025 fires (≥10 submission exports by a single actor within window)

Severity: High — potential insider data exfiltration

First 15 minutes: 1. SOC dashboard Tickets tab auto-creates ticket (per existing SOC rules, level≥10) 2. Identify actor: query audit_log filtered on action='submission_exported' for last 1h 3. Validate: is this a known admin doing legitimate export? Or unexpected? 4. If unexpected: revoke actor's forms portal access immediately

Investigation:

SELECT occurred_at, actor_username, actor_ip, details, request_id
FROM audit_log
WHERE action = 'submission_exported'
  AND occurred_at > NOW() - INTERVAL '24 hours'
ORDER BY occurred_at DESC;

- Cross-reference actor's recent auth_success / auth_failure events - Check if actor's Okta session came from unexpected IP or geo

Containment: - Revoke actor's Okta session: Okta Admin → Users → [actor] → Clear user sessions - Set users.is_active = false for the actor in forms DB - If compromise confirmed: force Okta password reset + MFA re-enrollment

Eradication: - Identify what was exported: details->>'submission_ids' in audit_log - Determine what data was in those submissions via decrypt (admin-only, audited) - Notify affected form owners per data classification policy

Recovery: - If data is recoverable (e.g., only staff names exposed internally): close with notification - If externally exposed: invoke DR-01 Data Breach playbook

Lessons learned: - Adjust Wazuh rule 100025 threshold if false-positive threshold was wrong - Consider rate-limiting /api/submissions export endpoints

FP-IR-03 — Decryption failure surge (Wazuh rule 100024)¶

Trigger: Wazuh alert 100024 fires repeatedly (multiple decrypt_failure events in audit_log)

Severity: Critical — possible KMS key rotation issue OR data tampering

First 15 minutes: 1. Check KMS key rotation state: gcloud kms keys versions list --key=gpus-forms-dek-wrapper --keyring=gpus-forms --location=us-central1 2. If a new key version was created recently (< 24h): rotation-related, benign-probable 3. If no recent rotation: possible tampering or DB corruption — escalate

Investigation: - SELECT occurred_at, target_id, details FROM audit_log WHERE action='decrypt_failure' ORDER BY occurred_at DESC LIMIT 100; - Which submission IDs fail? Isolated or widespread? - Query submission_fields: do kms_key_version values match current KMS state? - Check Cloud KMS audit logs: gcloud logging read 'resource.type=cloudkms_cryptokey AND protoPayload.methodName=~"Decrypt"'

Containment: - If key version mismatch: the old key version may have been auto-destroyed. Check gcloud kms keys versions list for DESTROYED state - If so: destroyed key version = affected submissions are unrecoverable. Identify scope. - If DB corruption: restore from latest backup to gpus-forms-db-recovery clone, compare submission_fields rows

Eradication: - If key issue: restore destroyed key version if still in 24h grace period (gcloud kms keys versions restore) - If corruption: restore affected rows from backup into production

Recovery: - Verify roundtrip decrypt on restored data: crypto.py roundtrip_test() endpoint - Notify submitters if their data needs re-submission (only if un-recoverable)

Lessons learned: - KMS rotation policy review - Consider per-submission DEK escrow for disaster scenarios (trade-off with security)

FP-IR-04 — Cloud SQL compromise / suspected unauthorized access¶

Trigger: Anomalous queries in Cloud SQL audit logs, unexpected schema changes, or Wazuh correlation alerts

Severity: Critical

First 15 minutes: 1. Disable forms backend traffic: gcloud run services update gpus-forms-backend --region=us-central1 --no-traffic 2. Capture current Cloud SQL state: gcloud sql backups create --instance=gpus-forms-db --description="IR-FP-IR-04-$(date +%s)" 3. Review Cloud SQL connection logs: gcloud logging read 'resource.type=cloudsql_database AND protoPayload.authenticationInfo.principalEmail!~"gpus-forms-backend"' 4. Check for unexpected IAM database users: gcloud sql users list --instance=gpus-forms-db

Investigation: - Compare current schema against expected (9 RLS policies, 12 tables, 4 roles): run the Q1-Q4 verification queries from Phase 1 Step 6 - Check audit_log for admin_reload_yaml or unexpected actions - Check if postgres built-in user has been password-set (signals someone elevated): SELECT rolname, rolvaliduntil FROM pg_roles WHERE rolname='postgres';

Containment: - Rotate postgres built-in user to throwaway random (same cleanup pattern from Phase 1 Step 6) - Revoke any unexpected IAM database users - Rotate the backend service account key (there isn't one today — we use ADC — but if keys were created, destroy them) - Re-verify RLS policies are intact; re-apply 002_rls.sql resume file if tampered

Eradication: - If data was altered: restore from pre-incident backup clone - Verify integrity by sampling decrypt roundtrip on submissions from the affected window

Recovery: - Re-enable traffic after all verifications pass - Post-mortem: how did attacker get in? Review VPC firewall rules, private IP exposure, service account key hygiene

Lessons learned: - Add Cloud SQL admin-activity alerts to Wazuh ruleset if not present

FP-IR-05 — KMS key compromise¶

Trigger: Cloud KMS key version showing unexpected decrypt calls, or compromise suspected

Severity: Critical — all submission data encrypted with that key version at risk

First 15 minutes: 1. Disable affected KMS key version: gcloud kms keys versions disable <version> --key=gpus-forms-dek-wrapper --keyring=gpus-forms --location=us-central1 2. Disable forms backend: gcloud run services update gpus-forms-backend --region=us-central1 --no-traffic 3. Capture KMS audit logs for the compromise window

Investigation: - gcloud logging read 'resource.type=cloudkms_cryptokey AND resource.labels.key_ring_id=gpus-forms' --freshness=7d - Identify all Decrypt calls + their principals - Cross-reference submission_fields rows with matching kms_key_version — these are the affected submissions

Containment: - Create new KMS key version: gcloud kms keys versions create --key=gpus-forms-dek-wrapper --keyring=gpus-forms --location=us-central1 - Update config.py KMS_KEY if key rotation behavior changed - Redeploy backend pointing at new version

Eradication: - Re-encrypt all affected submissions under new key version (batch job — requires admin-level process, NOT automatic) - Once all migrated, destroy compromised key version (30-day grace period)

Recovery: - Verify roundtrip on re-encrypted submissions - Re-enable traffic

Lessons learned: - Audit who had access to the key; review IAM bindings on gpus-forms-dek-wrapper - Consider HSM-backed keys if this recurs (increased cost)

References¶

Existing IRP: mkdocs-portal/docs/response-plans/irp.md
Tabletop playbooks: mkdocs-portal/docs/response-plans/tabletop-playbooks.md
SOC auto-ticketing thresholds: SOC dashboard Tickets tab
Cloud SQL IAM DB user recovery (if postgres password lost): recreate via gcloud sql users set-password postgres --instance=gpus-forms-db --password=<new> with project owner credentials