Skip to content

GPUS-IT Priorities

Classification: CONFIDENTIAL — Internal Use Only Document: priorities/gpus-it-priorities.md · v1.17 · 2026-07-24 · GPUS-IT Owner: Rajesh Chhetry · Review cadence: weekly (Fridays) + on every new initiative

Purpose

Living tracker of active and queued initiatives for the GPUS-IT infrastructure program. This document supersedes scattered priority notes across memory and Cowork; the canonical order of work lives here.

Rules of engagement:

  • Every initiative lists its gating dependencies — don't start work that is gated on unfinished prerequisites.
  • Every new asset added by an initiative must ship with its matching IR runbook (rb-00N-*.md), DR procedure update in drp.md, and red/blue drill entry in tabletop-playbooks.md + blue-team-drills.md. Infra docs without runbooks are not considered "done".
  • Status values: Planned · In Progress · Blocked · Done · Deferred
  • When an item completes, move it to Completed with the completion date and delete it from the active queue.

Sequencing

Forms portal is LIVE in production as of 2026-07-23 (go-live submission 28bd1ebf; routing worker f9a114b on MAPLE; override OFF; all 6 ingest addresses verified delivering; HappyFox tickets opening). Phases 2.5(a)–(d) are DONE and the broken-access-control remediation is complete. The forms program has shifted from build to cutover + hardening.

The order is now: cutover forms, harden forms, then Meraki, then WDC foundation.

  1. Forms cutover (T1, the only hard external deadline). Legacy forms portal (forms.us.gl3) retires 2026-08-01 — parallel running until then. The staff notice (email + Slack) and legacy decommission must land before that date. NOTE: Aug 1 is a Saturday — flag the cutover date for reconsideration (an LDAP deprovisioning gap argues for cutting over sooner, not later).
  2. Forms hardening (T1). Detection pipeline (blocks the drill program), remaining security hardening, content/data fixes, dedicated Okta forms app, 2.5(e) HappyFox API path, and portal propagation. Documentation set and the security exercise program (SQLi tabletop + blue/red drills) are gated on the detection pipelines existing.
  3. Meraki (T2) — Sedita fix + Meraki SSO, then fold Meraki into status/SOC/MkDocs coverage.
  4. WDC foundation (T3) — ESXi + Synology NAS inventory, cleanup, monitoring, and documentation. Deferred intentionally: the hypervisor + NAS are stable enough today that finishing the forms cutover + hardening is higher-value.

Everything else (SOC Ticketing, Vendor Access, MySQL decom, status site automation) slots in after WDC foundation unless a specific gating dependency inverts the order.

Summary

Tier Initiative Status Target
T1 Forms portal LIVE in production — go-live 2026-07-23 (sub 28bd1ebf); routing worker f9a114b on MAPLE, override OFF, all 6 ingest addresses verified delivering, HappyFox tickets opening Done 2026-07-23
T1 Broken-access-control remediation — Phase-1 decrypt/list/submit endpoints removed; require_role fail-closed; resolve_user least-privilege; IDOR ownership checks on Phase-2 endpoints. NO evidence of exploitation (empty audit_log + empty users table, independently corroborated) Done 2026-07-23
T1 Forms 2.5(a)–(d) backend + routing worker — submit/attachment/finalize wired, subject-template audit-persist gate resolved, GAP-1 renderer fix + template reconciliation landed Done 2026-07-23
T1 Forms cutover → forms.greenpeace.us — staff notice (email+Slack), legacy forms.us.gl3 decommission, in-flight handling, redirect-or-dark. Only hard external deadline. LDAP deprovisioning gap argues for cutting over sooner In Progress 2026-08-01 (Sat — reconsider)
T1 Detection pipeline — audit_log→SOC + Cloud Run→Wazuh log sinks (neither exists); Wazuh rules 100026–100029 inert by starvation, not config. Blocks the exercise program Planned 2026-08
T1 Security hardening — post-go-live — retire legacy auth.py authz primitive; narrow backend-SA project-level over-grants; Cloud Armor/WAF (arch change + cost case); /metrics + /health/deep exposure; CORS *; per-instance rate limiting; MAC field validation Planned 2026-08
T1 Content / data fixes — <% = Note => in Finance Termination template rendering literally in prod; live-DB legacy-tag sweep; DB↔repo drift root cause (HIGH); Insurance checkbox unanswerable-as-No; HR Termination authoring TODO; footer sweep; DSAR/erasure path; attachments retention lock Planned 2026-08
T1 Credential-rotation control never closed out (compliance drift) — the quarterly forms-portal-credential-rotation-quarterly control emits records but is never back-filled: 3 ACTION-NEEDED carried since Q2 (KMS rotation timestamps, Cloud SQL backup verification, HappyFox credential rotation) + an MFA-enforcement flag; possibly two consecutive quarters with no live console verification. Same class as the contract-vs-code and DB↔repo drift findings — artifacts asserting a state nobody verified. PRE-ANNOUNCEMENT GATE: confirm Okta MFA enforcement before the staff notice goes out — forms authz now rests entirely on Okta identity and the announcement leads with "sign in with Okta". Planned MFA: before cutover notice
T1 Okta — dedicated forms app + group — requested from Conan 2026-07-23; then verify SPA redirect URI, aud validation, preview-tenant test, client-ID cutover In Progress 2026-08
T1 2.5(e) HappyFox API — queue names from API, ticket-audit migration, per-queue double-ticket deconfliction; API preferred over email-ingest (greenpeace.us DMARC p=none; help.greenpeace.org no DMARC → forged ticket plausible); ingest sender restrictions unverified Planned 2026-08
T1 Portal propagation — inventory.yaml reflects live forms state; coverage + validate_portal_presence pass; status + SOC show forms LIVE (derived, not hardcoded); verify live pages render it Planned 2026-08
T1 Okta cleanup — remove localhost redirect URIs (separate from the dedicated forms app above) Planned 2026-08
T1 Forms portal Phase 1.5 — legacy data migration Planned 2026-08
T1 Forms portal Phase 1.6 — ON CONFLICT refactor Planned 2026-08
T1.8 Forms documentation set (mkdocs) — threat model; OWASP ASVS L2; PCI-DSS applicability (out of scope, no CHD); OWASP→NIST 800-53→MITRE mapping; forms IR runbook + DRP entry; BAC security-finding record; governance finding; 2.5(d) as-built; field-exposure matrix regen; mitre-attack/threat-vectors/pentest-schedule/calendar updates. Depends on the assessment Planned 2026-08
T1.9 Forms security exercises — SQLi tabletop (60m), blue-team detection drill (90m), red-team simulation (90m). GATED on detection pipelines Blocked 2026-09
T1 α ClamAV scan worker — Commit 3 (hardening + hygiene, 8 items) Filed 2026-08
T2 Meraki cleanup — Sedita site + Meraki SSO Planned 2026-06
T2 Meraki integration — status/SOC/MkDocs coverage Planned 2026-06
T3 WDC foundation — ESXi inventory & cleanup Planned 2026-07
T3 WDC foundation — Synology NAS inventory & cleanup Planned 2026-07
T3 GCP terraform tree has no VCS — dedicated session: secrets review of tfvars/tfstate, .gitignore design, git init, push to private remote (CSR), import state (supersedes WDC VPN/route drift framing) Filed 2026-06
T3 VPN cold-start packet loss — diagnosis + fix (Mac → GCP private subnet path drops on cold start; 2nd incident in 7 days) Filed 2026-06
T3 IAP-as-break-glass — IAP-to-MAPLE 4003 confirmed host-level 2026-06-10 (not IAP edge); fix VM-side so IAP works as break-glass path when VPN/SSH down Filed 2026-06
T3 Cloud Scheduler missed-tick on cold-start (sweep.clean missed 05:45 UTC 2026-05-21; investigate scheduler retry config vs worker min-instances tradeoff) Filed 2026-06
T3 SKY portal-backup cron died ~2026-04-17 — portals/ snapshots stop there (server backups current). Restore the cron; backfill the gap Filed 2026-08
T3 2.5(e) PRECONDITION — HappyFox ingest/API deconfliction. G4.4 proved email_template legs landing in HappyFox-watched inboxes (e.g. gpus-it-support@) auto-open tickets via email-to-ticket (#USITS00357259, 2026-06-12; worker dispatched no happyfox action — verified). Once 2.5(e) dispatches via API, forms with both action types against the same queue will DOUBLE-TICKET. Deconflict per-form email recipients vs API destinations before 2.5(e). Context: org moved to API because HappyFox blocks greenpeace.us email — the ingest path predates that and is still live. Filed feeds 2.5(e)
T4 SOC Ticketing tab Planned 2026-08
T4 Vendor Access Portal (replace SFTP) Planned 2026-08
T5 Status site automation — Phase A (live Cloud Run list) Planned 2026-09
T5 MySQL decommission Planned 2026-09
T5 Status site automation — Phase B (BigQuery billing export) Planned 2026-10
T5 Status site automation — Phase C (per-service cost attribution) Planned 2026-11
T5 T5-EXPANDED — Portal static-debt retirement (P0–P4, per 2026-06-10 audit; forms Gate 5 now landed → unqueued; P0 truth-fixes may interleave with forms hardening) Filed 2026-Q3

Recently completed forms/α items (see Completed table): forms 2.5(a)–(d), routing worker (Gate 4/5), BAC remediation, T1.7 SPA submit-gate, α ClamAV Commits 1 + 2.

T1 — Forms: cutover & post-go-live hardening

Forms portal is LIVE in production as of 2026-07-23. Go-live submission 28bd1ebf; routing worker f9a114b on MAPLE with override OFF; all 6 ingest addresses verified delivering; HappyFox tickets opening on real submissions. Phases 2.5(a)–(d) DONE (see the phase records further down this section, kept as history). The active T1 work is now cutover (the only hard external deadline) plus the hardening workstreams below. The phase-history sub-sections (Phase 2, 2.5(a)–(e), UX gaps, etc.) follow as completed records.

Cutover to forms.greenpeace.us — the only hard external deadline

  • Status: In Progress. Legacy forms portal (forms.us.gl3) retires 2026-08-01; new and legacy run in parallel until then.
  • ⚠ Date flag: 2026-08-01 is a Saturday. Flag for reconsideration — a weekend decommission means no staffed coverage if a rollback or in-flight-submission issue surfaces. Weigh against the LDAP argument below, which pushes the other way (sooner).
  • Items:
    • Staff notice (email + Slack). Operational, independent of every other workstream — has no technical dependency and must go out before Aug 1. Do not let it wait on hardening.
    • Legacy decommission of forms.us.gl3 — plan the teardown; decide redirect-or-dark for the old hostname.
    • In-flight submission handling — drain/route anything submitted to the legacy portal during the overlap window.
    • Duplicate-ticket ambiguity during overlap — both portals feed the same ingest addresses, so a form submitted on both (or mid-migration) can double-open tickets. Deconflict before/during the overlap (ties into 2.5(e) per-queue deconfliction below).
    • LDAP deprovisioning gap — argues for cutting over SOONER. The Okta→LDAP sync keeps breaking, so a disabled account may still authenticate to the legacy portal. Every day of parallel running is a day a deprovisioned user retains legacy access. This is the security case for an earlier cutover, in tension with the Saturday-date concern above.

Detection pipeline — blocks the drill program

  • Status: Planned. Gating dependency for the T1.9 exercise program — there is no point running blue-team detection drills against detections that cannot fire.
  • Items:
    • audit_log → SOC pipeline — does not exist. Forms audit events never reach SOC. (Overlaps the long-standing T1.6 observability workstream; this is the concrete blocker for drills.)
    • Cloud Run → Wazuh log sink — does not exist. Wazuh rules 100026–100029 are inert by starvation, not by misconfiguration — the rules exist but no forms/Cloud Run events are being fed to them.
    • Verify Wazuh rule bodies + ossec.conf on MAPLE — confirm the rule definitions and agent config are actually what we think before wiring the feed.
    • Routing-worker systemd unit has an empty SyslogIdentifier — fix so the worker's logs are attributable in the journal/sink (prerequisite for the Cloud Run→Wazuh feed to be useful).

Security hardening — post-go-live (remaining)

The broken-access-control remediation is DONE (see the dedicated record below). These are the remaining hardening items surfaced during/after go-live:

  • Retire legacy auth.py authz primitive (Step 4). Migrate /api/admin/reload to auth_v2; the users table becomes login-audit only. (Completes the auth-module consolidation begun in Phase 2.1 item 5.)
  • Backend service-account over-grants. The backend SA holds project-level roles/cloudkms.cryptoKeyEncrypterDecrypter + project-wide roles/storage.objectAdmin. Narrow to the specific key and bucket scope (least privilege).
  • Cloud Armor / WAF — NOTE this is an architecture change, not a policy toggle. forms.greenpeace.us is a direct Cloud Run domain mapping today; adding Cloud Armor requires standing up a new external HTTP(S) load balancer + serverless NEG in front. Treat as an architecture change + cost case, not a quick enablement.
  • /metrics is unauthenticated — leaks per-form submission volumes. Gate it.
  • /health/deep exposes DB + KMS status publicly — reduce to an authenticated/internal probe.
  • CORS origins="*" — tighten to the known SPA origin(s).
  • Rate limiting is memory:// per-instance — the effective limit multiplies by instance count (so the real ceiling is N× the configured value). Move to a shared backend or account for instance count.
  • MAC field accepts malformed input — add validation.

Broken-access-control remediation — DONE (2026-07-23)

  • Status: COMPLETE. Recorded here as a closed security finding; the full finding record (finding + remediation + no-exploitation conclusion) is a T1.8 documentation deliverable.
  • What was fixed:
    • Phase-1 decrypt / list / submit endpoints removed.
    • require_role made fail-closed.
    • resolve_user moved to least-privilege.
    • IDOR ownership checks added on the Phase-2 endpoints.
  • No evidence of exploitation — corroborated independently by two facts: the audit_log is empty of any decrypt/view events, and the users table is empty. Two independent signals both consistent with "never exploited."

Content / data fixes

  • <% = Note => legacy placeholder surviving in the Finance Termination template — renders literally into production tickets. Highest-visibility content bug post-go-live.
  • Live-DB sweep for other legacy tags — a repo sweep provably misses drift (see DB↔repo root-cause item), so the sweep must run against the live database, not the repo.
  • DB↔repo drift root cause — HIGH IMPORTANCE. If the repo is not the authoritative source for templates/forms, then the gap-2 sweep, the field-exposure matrix, and the CI leak-gate are all validating the wrong artifact. Root-cause which store is authoritative before trusting any repo-based check.
  • Insurance checkbox: required with only "Yes" — unanswerable as "No" (a required field the user cannot legitimately decline). Fix, and sweep for the same pattern across other forms.
  • HR Termination template contains an unfinished authoring TODO line — remove/complete before it renders to a recipient.
  • forms.us.gl3 → forms.greenpeace.us footer sweep — replace stale legacy-hostname footers.
  • No DSAR / subject-erasure path — there is no implemented data-subject-access-request or erasure flow; the retention purge executor is unlocated; the attachments bucket retention (2555d) is still UNLOCKED. Privacy/retention gap to close.

Okta — dedicated forms app + group

  • Status: In Progress — dedicated forms app + group requested from Conan 2026-07-23.
  • Then: verify the SPA redirect URI, aud (audience) validation, run a preview-tenant test, and perform the client-ID cutover to the new app.
  • Related but separate: the existing "remove localhost redirect URIs" cleanup (Phase 2.1 / Okta cleanup section) is a different task from standing up the dedicated app.

2.5(e) — HappyFox API integration

  • Status: Planned. Native HappyFox API ticket creation, distinct from today's live email-ingest path.
  • Items:
    • Queue names from the API (stop hardcoding queue identifiers).
    • Ticket-audit migration — persist HappyFox ticket IDs / outcomes into the audit trail.
    • PER-QUEUE double-ticket deconfliction — a single form routing to N queues risks doubling a ticket across all N once both the email-ingest and API paths are active. This is the same hazard as the cutover overlap duplicate-ticket item; solve once, per queue. (See the T3 "2.5(e) PRECONDITION" row for the G4.4 evidence.)
    • API vs email-ingest decision — security prefers API. Rationale: greenpeace.us is DMARC p=none (vs .org's p=reject), and help.greenpeace.org has no DMARC record at all, so a forged portal-looking email ticket is plausible on the ingest path. The authenticated API path removes that spoofing surface.
    • HappyFox ingest sender restrictions unverified — confirm whether the ingest inboxes actually restrict senders (if not, the spoofing risk above is live today).

Portal propagation

  • Status: Planned (residual of forms Gate 5). The standing rule applies: portal content must be LIVE or DERIVED, never hardcoded, and must render from sources, not merely have endpoints exist.
  • Items:
    • inventory.yaml reflects the live forms state.
    • Coverage + validate_portal_presence pass.
    • status + soc portals show forms LIVE — derived, not hardcoded.
    • Verify the live pages actually render it (the audit standard: endpoints existing is not enough; the frontend must display the derived value).

Phase 2 React + Okta (forms frontend SPA)

  • Status: Phase 2 frontend SPA: COMPLETE 2026-04-27. Live on forms.greenpeace.us. Backend rev gpus-forms-backend-00033-bfx, frontend rev gpus-forms-frontend-00003-74f. 28 forms render with editorial typography. Auth pipeline (Okta OIDC PKCE → JWKS → /api/forms) fully verified end-to-end.
  • Goal: Ship gpus-forms-frontend as React SPA with Okta OIDC PKCE auth, calling gpus-forms-backend with a validated JWT bearer.
  • Gating: None — gpus-forms-backend is live on Cloud Run, forms.greenpeace.us resolves, Okta Production cutover complete (2026-04-23).
  • Deliverables:
    • forms-frontend/ in gpus-infra-portals repo
    • JWT validation middleware on gpus-forms-backend (and on status / security / soc backends — Phase 2 is org-wide, not forms-only)
    • Cloud Run deploy + Cloud Build trigger wired
  • Asset docs required: Update iar.md with forms-frontend service; no new IR runbook needed (covered by existing portal runbooks); blue-team drill entry for "forged/expired JWT rejected" in blue-team-drills.md.
  • Phase 2 follow-up: FieldRenderer pulldown lookup — COMPLETE 2026-04-28 (commit 5512d8e). Phase 1's /api/forms/<id> serializer now returns id/label/pulldown_id (matching contract.ts). New endpoint GET /api/pulldowns/<name> added with PulldownResponse shape. Backend rev gpus-forms-backend-00034-8tn.

Phase 2.5 — Phase 2 backend wire-up

  • Discovered 2026-04-28: the Phase 2 SPA has been live but Phase 2's three submission endpoints (POST /api/submissions, POST .../attachments, POST .../submit) are unwired stubs in routes_phase2.py. They return fake UUIDs and a hardcoded 'TBD@greenpeace.us' literal. Submitting a form via the SPA appears to work to the user but persists nothing — no DB write, no GCS upload, no email/HappyFox call. The well-formed routing model exists (actions table populated by the MySQL migrator with per-form action_type + destination + template_id) but no submit-time code reads it. There is also no mailer module in the backend at all.
  • Implementation scope:
    1. Wire create_submission (routes_phase2.py:102) to write to the submissions table with KMS envelope encryption for non-searchable fields, returning real UUIDs. Pattern reference: routes/submissions.py:37 (Phase 1's working submit handler).
      • STATUS: COMPLETE 2026-04-30
        • Commit 1 SHIPPED 2026-04-29: 07cb75c (auth_v2 User.username via preferred_username claim, rev gpus-forms-backend-00036-89b).
        • Commit 2 SHIPPED 2026-04-30: 10bcd0d (design doc to mkdocs-portal/docs/architecture/forms-phase2.5a-design.md).
        • Commit 3 SHIPPED 2026-04-30: 83f85cb (routes_phase2.py create_submission wire-up, rev gpus-forms-backend-00038-5kg).
        • Verified end-to-end 2026-04-30: API contract response shape, DB persistence (encrypted fields + audit log), MAPLE-side inspection of test submission 0bac9326-4968-477b-8d10-d7d6f457e2a8.
        • Phase 2 SPA submissions now ACTUALLY persist (was theater since Phase 2 cutover).
    2. Wire upload_attachments (routes_phase2.py:128) to GCS bucket gpus-forms-attachments with ClamAV scan and attachment row inserts.
      • STATUS: COMPLETE 2026-05-08 (β closeout — see architecture/forms-phase2.5b-cleanup-closeout.md)
        • 2.5(b) handler shipped 2026-05-08: e17dacb (rev gpus-forms-backend-00041-mcf), three-layer verification PASS on test submission 999cf0cc-… (GCS object + attachments row + audit_log row, all consistent to the microsecond).
        • 2.5(b.cleanup) shipped 2026-05-08: 28964c0 (rev gpus-forms-backend-00043-d9q), 4-source MIME/size truth (prod env vars, Config, module-level shadows, _schema.yaml hint) collapsed to Config.ATTACHMENT_MAX_BYTES + Config.ATTACHMENT_ALLOWED_MIME as single source. Production env vars MAX_UPLOAD_BYTES + ALLOWED_MIME_TYPES removed.
        • Schema migration forms-backend/schema/002_add_submission_deleted_action.sql documents submission_deleted audit_action enum value (added in production during β step 8 to enable orphan-intent cleanup).
        • ClamAV scan handled in Phase 2.5(b.2) — see dedicated section below, COMPLETED 2026-05-19. At β time all attachments landed with clamav_status='pending'; partial index idx_attachments_clamav was already in place for the scanner.
        • Verification gap accepted: end-to-end positive case on rev 00043-d9q blocked by T1.7 SPA bug (silent attachment drop). Pre-cleanup positive cases (5c57b2e6 docx, 62d2fc73 xlsx) plus mechanical-substitution diff correctness accepted as evidence of backend behavior. Rationale in closeout doc.
    3. Wire finalize_submission (routes_phase2.py:151) to:
      • STATUS: COMPLETE — SHIPPED TO PRODUCTION 2026-07-23 (go-live). The routing worker is live: rev f9a114b on MAPLE, override OFF, all 6 ingest addresses verified delivering, HappyFox tickets opening on real submissions (go-live submission 28bd1ebf). Gates 3, 4, and 5 all landed. Locked decisions (from v0.2) as shipped:
        • Transport/shape: B-iii — MAPLE-resident routing worker, Pub/Sub pull subscriber, sends via localhost:25 Postfix reusing the report_mailer.py pattern. Zero new secret. First non-serverless service in the rebuild → carries its own systemd unit + monitoring + IR/DR/drill obligations (see the routing-worker SyslogIdentifier note under Detection).
        • DB role: reuse forms_app (no new role; RLS USING(TRUE) makes the GATE-4 state-coverage work N/A).
        • Attachments: signed-URL delivery in body (inline MIME deferred).
        • Interim body: minimal plaintext, NOT gated behind 2.5(d).
        • Audit: lean — migration 011_routing_audit_actions.sql (4 enum values) + 012 (submission_finalized), both live.
      • Gate close-out (2026-06-10 → 2026-07-23): Gate 3 (C1 GET, C2 routing_result finalize marker, C3 full predicate at finalize, C4 α publishes unconditionally, C5 = migration 012), Gate 4 (routing worker), and Gate 5 (propagation: inventory + coverage-standard portal-propagation section + status Services section + SOC alerting + ops triad + α backfill) all DONE. Residual Gate-5 propagation verification is carried forward as the Portal propagation workstream below (confirm live pages actually render forms LIVE, derived not hardcoded).
      • SELECT actions FROM actions WHERE form_id = <...> ORDER BY action_order
      • For each action, branch by action_type:
        • happyfox_template → render via templates table, POST to HappyFox API (secrets already in Secret Manager: gpus-forms-happyfox-api-key, gpus-forms-happyfox-auth-code)
        • email_template / email_raw → render template, SMTP send via Postfix on MAPLE (currently no SMTP client in backend)
      • Aggregate routing_result into submissions.routing_result JSONB.
    4. Build template render layer (Jinja2 against templates(id, body)).
    5. Build HappyFox API client (configures the integration that SOC dashboard memory #20 noted as "not configured").
    6. Decide policy on email_template / email_raw actions: keep, drop, or redirect through HappyFox? (Per Rajesh: HappyFox API is the operational current path because Google's stricter email policies broke the email-to-HappyFox flow.)
  • Estimated effort: ~~multi-session work~~ DONE. Sequence ran (a)→(b)→(c)→(d) with the routing/HappyFox path shipped at go-live 2026-07-23.
  • Gating: completed the "forms portal actually works" milestone. The 2.5(e) HappyFox API client (native ticket creation, distinct from today's email-ingest path) is now its own workstream below. The SQLi tabletop drill is unblocked functionally but is now GATED on the detection pipeline existing (no point drilling detection that can't fire) — see T1.9.

Phase 2.5(b.2) — α ClamAV scan worker

  • Status: Filed 2026-05-19. COMPLETED 2026-05-19 (5 migrations 004-008 shipped — enum + SQL user + table grants + tight scanner RLS + USING widen + sequence USAGE; pipeline verified end-to-end: 07c3c9e0 fixture + 4a0f5bd5 real upload both clean with audit rows; worker rev gpus-forms-clamav-worker-00002-dwx on Cloud Run, 2Gi, min-instances=0).
  • What shipped: new Cloud Run service gpus-forms-clamav-worker (Pub/Sub-push on gpus-forms-attachments OBJECT_FINALIZE → claim → clamscan → verdict to attachments + audit_log). Scoped Terraform (SA + IAM + topic + DLQ + OIDC push sub; VPN drift untouched, see T3 row). CSR Cloud Build triggers (push + weekly sigrefresh). Design doc alpha-clamav-worker.md v1.2.
  • α.1 / Commit-2 follow-ups: shipped 2026-05-20 in Commit 2 — see ### Phase 2.5(b.2) — Commit 2 immediately below.

Phase 2.5(b.2) — Commit 2 (Slack + GCS tag + DLQ + sweep)

  • Status: COMPLETED 2026-05-20 (worker rev gpus-forms-clamav-worker-00010-gr7 on :c9c2016; migrations 009 + 010 shipped; Terraform 6 resources applied in clamav-worker.tf; EICAR test verified the full infected pipeline end-to-end at 16:45 UTC).
  • What shipped:
    • Slack-post for infected verdicts — placeholder-tolerant fetch from Secret Manager (gpus-forms-clamav-slack-webhook), lazy module-global cache, cold-start dance documented; verified class=wired on first call during EICAR.
    • GCS quarantine metadata tag (quarantined=true, quarantine_reason=<sig>, quarantined_at=<iso8601>) — best-effort with try/except per C4 ordering (tag first, Slack second; audit row is the system of record).
    • /dlq-alert route + DLQ push subscription on gpus-forms-attachment-uploaded-dlq — closes the design §10 gap surfaced at Commit 1 GATE 4. Always returns 200 (no DLQ-of-DLQ).
    • /sweep-stuck route + Cloud Scheduler tick every 15 min — closes α.1 stuck-scanning recovery. Atomic UPDATE … RETURNING, one audit row per flipped id with revision_at_sweep from K_REVISION for deploy correlation, Slack only on flips > 0.
    • Cloud Scheduler API enabled on gpus-infra; clamav-scheduler@gpus-infra.iam SA wired with run.invoker on the worker (resource-scoped) + cloudbuild.builds.editor on the project.
    • Weekly sigrefresh scheduler job (Sun 02:00 UTC) fires the existing gpus-forms-clamav-worker-sigrefresh build trigger — closes the README's "weekly auto-refresh — NOT yet wired" follow-up from Commit 1.
  • Verification: EICAR test at 16:45 UTC. Full pipeline green:
    • event.received → claim.ok → clamscan → INFECTED signature=Eicar-Test-Signature → quarantine.tag.ok → slack.webhook.resolved class=wired (first fetch — lazy init confirmed) → Slack post landed in #us-soc-alerts → audit row attachment_scanned_infected written.
    • All 5 verification checks (worker log, GCS metadata, attachments row, audit_log row, Slack channel visual) passed.
    • Bonus signal: the */15 sweep tick fired naturally at 16:45:03 UTC during the test window, logging sweep.clean count=0 — gate 2d wiring proven live end-to-end without manual stimulus.
    • Test fixture (1 submission + 1 attachment + 1 audit row + 1 GCS object) cleaned up post-validation; attachments now back at pre-test state (clean=2, pending=3).
  • α.1 / Commit 3 follow-ups (hardening / hygiene — NOT Commit-2 blockers):
    • Tighten cloudbuild.builds.editor to per-trigger IAM on just gpus-forms-clamav-worker-sigrefresh (currently project-wide — over-broad blast radius).
    • Rename SLACK_PLACEHOLDER_PREFIX_OK → SLACK_VALID_PREFIX (constant name misleads; logic is correct).
    • Normalize _sweep_stuck audit INSERT to use the _AUDIT_ACTION dict (currently hardcodes the enum value as a string literal; PG casts implicitly so it works, but inconsistent with _record).
    • Sigrefresh build trigger needs --included-files=gpus-forms-clamav-worker/** filter — currently fires on every push to main regardless of path, racing with the main trigger and burning a build slot.
    • forms-backend/schema/ 002 ordinal collision (002_rls.sql + 002_add_submission_deleted_action.sql) — rename the second to a higher ordinal.
    • Backfill drain: 3 remaining pending attachments (real user uploads pre-dating Commit-1 worker deployment) still need scanning — carried over from Commit 1; safe to drain via the established byte-correct re-fire pattern, OR let /sweep-stuck pick them up if their uploaded_at is in the stuck window.
    • Commit the ~/terraform/gpus-infra/terraform/clamav-worker.tf Commit-2 additions to that repo (applied to GCP but the .tf change is local-only).
  • T3 candidates filed during this arc (separate from Commit 3):
    • VPN cold-start packet loss — 2nd incident in 7 days; routing to private-IP Cloud SQL (10.34.0.0/24, Private Services Access) failed mid-session.
    • IAP-to-MAPLE 4003 backend-fail — only matters as a backup access path when direct SSH is also down, but should still resolve.

Phase 2.5(a) design doc correction

  • Status: CLOSED 2026-05-07 via 2de57c9. Four corrections applied to mkdocs-portal/docs/architecture/forms-phase2.5a-design.md to match shipped code (commit 83f85cb):

    • _audit_v2 helper signature (kwargs + body)
    • allow-list deny audit action ("auth_failure" not "submission_denied")
    • SubmissionField attachment (session.flush + submission_id, not relationship)
    • second _audit_v2 call site (consistency fix)

    Plus a process-note appendix appended to the doc capturing the read-first-discipline lesson. - Discovered: 2026-04-30 during Phase 2.5(a) Commit 3 implementation. - Context: The design doc at mkdocs-portal/docs/architecture/forms-phase2.5a-design.md was committed in 10bcd0d and contained 3 bugs in the helper specs that Code caught before Commit 3 shipped: 1. _audit_v2 helper signature — design said actor=actor kwarg; correct is actor_username + 4 other Phase 1 fields (target_id, target_type, details, request_id, success). 2. Allow-list deny audit action — design said "submission_denied"; that value isn't in the audit_action ENUM. Correct is "auth_failure" with details={"reason": "not_in_allow_list"} and success=False — matches Phase 1's existing pattern. 3. SubmissionField attachment pattern — design used row.submission = submission (assumes ORM relationship that doesn't exist). Correct is Phase 1's session.flush() + submission_id=submission.id pattern.

Forms Phase 2 UX gaps

  • Discovered: 2026-04-29 PM during Phase 2.5(a) browser regression check.
  • Status: T1.5 sub-section is now FULLY CLOSED — items a/b/c/d all closed. Section can stay in the doc as a closed-finding record or be moved to a "Recently completed" archive at section author's discretion. Item (a) CLOSED 2026-05-07 via f31ec5f. Items (b), (c), (d) CLOSED 2026-04-30 via 2026-04-30 γ phase commits.
  • Priority order within sub-section: a (pulldown — user-visible blocker for form submission accuracy) → d (font — visible polish) → b, c (cosmetic).

  • Items:

    a. Pulldown regression — yes/no booleans — CLOSED 2026-05-07 via f31ec5f.

    Root cause: Flask `<string>` route converter (the default) does not match forward slash. Pulldown names containing `/` (e.g. "Grant Funded ? (Yes/No)", "Bargain Type (In/Out)", "Shipping Label/Box Required?") failed to route — Cloud Run URL processing decoded `%2F` back to `/` before route matching, splitting the path into multiple segments that no route consumed. Result: 404 → SPA renders "Could not load options".

Fix: change the `@bp.get` decorator from `"/pulldowns/<name>"` to `"/pulldowns/<path:name>"` — single character change.

Diagnosis chain used DB inspection (via MAPLE access pattern established 2026-04-30): hypothesis 1 (missing pulldown rows) was falsified by data showing all referenced `pulldown_name`s exist in the `pulldowns` table. Pattern noticed in failing data: all 7 failing names contained `/`.

    b. COST CENTER duplicate field — CLOSED 2026-04-30 via 6d9231a (resolved as a side effect of the NO_DATA_TYPES filter — duplicate was a divider/instructions row with that label).

    c. NOTESDIVIDER label leak — CLOSED 2026-04-30 via 6d9231a (forms-backend routes/forms.py filters NO_DATA_TYPES from form-detail serializer).

    d. Font darkness / contrast — CLOSED 2026-04-30 via f5c79e0 + e58dabc (forms-frontend .field-label color #4a4a44 — WCAG AA at ~9.6:1 contrast over --surface-warm).

Phase 2.1 — Phase 1 Production cutover completion

  • Goal: Finish the Okta Preview → Production cutover that only partially landed in Phase 1's auth/config layer. Phase 1 was originally built against the Okta Preview tenant; the Production cutover on 2026-04-23 left several stale references and shim layers in place. The 5 items below were surfaced 2026-04-27 during the forms-frontend smoke test and consolidate the remaining cleanup. None are user-visible bugs today; they are maintenance traps.
  • Gating: None — Phase 2 SPA is live; cleanup can land at any time.
  • Items (priority order):
    1. forms-backend/auth.py — replace audience=config.OKTA_AUDIENCE with audience=config.OKTA_CLIENT_ID. Remove the OKTA_AUDIENCE Cloud Run env var workaround. Estimated 1 commit, ~5 min.
    2. forms-backend/config.py — change OKTA_ISSUER = f"https://{os.environ.get('OKTA_DOMAIN', 'greenpeaceeu.oktapreview.com')}" to read OKTA_ISSUER directly with default https://greenpeaceeu.okta.com, matching auth_v2.py's pattern. Remove OKTA_DOMAIN env var afterward. Estimated 1 commit, ~10 min.
    3. forms-backend security headers — remove CSP/HSTS/X-Frame-Options/etc. from Flask middleware. nginx (forms-frontend/nginx.conf) is now the sole source for response-time security headers. The duplicate headers don't break anything but create maintenance traps. Estimated 1 commit, ~15 min.
    4. forms-backend CSP — any CSP that must remain in Flask should remove stale greenpeaceeu.oktapreview.com references. Replace with greenpeaceeu.okta.com. Stale since the 2026-04-23 Preview→Prod cutover. Estimated 1 commit, ~10 min.
    5. Consolidate forms-backend/auth.py and auth_v2.py into a single module once both stable. The _key_for_kid retry pattern shipped 2026-04-27 in auth.py (commit d0a7867) carries forward. The v2 module's signin/role pattern carries forward. Defer until after a few weeks of stable production traffic. Estimated 1 PR-sized change.

T1.9 — Forms security exercise program (SQLi tabletop + blue/red drills)

  • Status: BLOCKED — gated on the Detection pipeline (audit_log→SOC and Cloud Run→Wazuh log sinks must exist first; Wazuh rules 100026–100029 are inert by starvation today). No point running a blue-team detection drill against detections that cannot fire.
  • Goal: Exercise detection and response for SQL injection against the live forms backend.
  • Components: SQLi tabletop (60m) → blue-team detection drill (90m) against the live endpoint → red-team simulation (90m) adversarial test.
  • Deliverables: Updates to tabletop-playbooks.md and blue-team-drills.md; Wazuh (+ any WAF) rule tuning if gaps surface; executive summary.
  • Owner: Rajesh.

Phase 3 HappyFox integration (forms backend) — SUPERSEDED by 2.5(e)

  • Status: Superseded. The auto-ticket routing shipped at go-live via the routing worker's happyfox_template / email-ingest legs. Remaining native-API work is tracked as 2.5(e) — HappyFox API integration above (queue names from API, ticket-audit migration, per-queue double-ticket deconfliction, API-vs-ingest decision). Kept here for lineage.
  • Historical goal: Wire form submissions to auto-create HappyFox tickets per form's routing config; retry + DLQ on failed ticket creation; ticket ID written back to submission record.
  • Asset docs required (carried into T1.8): iar.md (HappyFox as integrated system); forms IR runbook "HappyFox API outage" + "ticket creation failure → DLQ drain"; blue-team drill entry for "spoofed HappyFox webhook".

Forms portal Phase 1.5 — legacy data migration

  • Goal: Migrate existing legacy-form submissions (where applicable) into the new schema.
  • Gating: Phase 3 HappyFox integration live (so migrated records route correctly on any post-migration edits).
  • Scope: One-time batch import; validate row counts, checksum fields, encrypted columns round-trip correctly; retain legacy source read-only for 90 days post-migration.

Forms portal Phase 1.6 — ON CONFLICT refactor

  • Goal: Refactor upsert logic in gpus-forms-backend to use proper ON CONFLICT clauses rather than check-then-insert race conditions.
  • Gating: Phase 1.5 complete (don't refactor insert paths mid-migration).
  • Scope: Submission insert, audit_log append, pulldown cache refresh.

Okta cleanup — remove localhost redirect URIs

  • Goal: Remove dev-convenience localhost redirect URIs from the Okta Production app (0oavvg1y33wTWFsmP417) once Cloud Run deploy of forms-frontend verifies.
  • Gating: Phase 2 Cloud Run deploy verified — met (forms live in production). Unblocked; can land at any time. Coordinate with the dedicated-forms-app cutover above so the URI cleanup targets the right app.
  • Done-when: Production app redirect URI list contains only https://*.greenpeace.us/* entries. Preview tenant (greenpeaceeu.oktapreview.com, client 0oadhpjktd5UfCMDm0x7) retained as dev fallback.

T1.6 — Forms portal SOC / observability integration

STATUS: NOT STARTED — gap surfaced 2026-05-08 during Phase 2.5(b) scoping discussion

Priority: P2 — should ship after Phase 2.5 functional work completes (2.5b/c/d/e), before SOC Tickets workstream (T4) starts. T4 will assume forms portal events flow into SOC; this workstream makes that true.

Context:

The forms portal (forms.greenpeace.us, gpus-forms-backend, gpus-forms-frontend, Cloud SQL gpus-forms-db) has been operationally invisible to SOC since Phase 1 cutover. While the rest of GPUS infrastructure (WDC servers SKY/RAIN/SUN/WIND, GCP VMs OAK/MAPLE/CEDAR) emits structured events into Wazuh + ELK + Prometheus and is visible across the 15 tabs of soc.greenpeace.us, forms portal emits zero events into that pipeline.

Existing forms-portal capabilities:

  • KMS envelope encryption for sensitive form fields (Phase 2.5a)
  • Audit log table in Cloud SQL with actor_username, actor_ip, target_id, request_id, success
  • IAM-protected Cloud SQL access via service accounts
  • Allow-list authorization on form submission

Observability gaps:

  • No SOC dashboard tab — the 15 tabs at soc.greenpeace.us don't include "Forms"
  • No Wazuh rule coverage for forms-portal events
  • No Cloud Logging → CEDAR/MAPLE pipeline for forms events (audit log lives only in Postgres, not exported)
  • No Prometheus metrics emitted from forms-backend
  • No alert routing for forms-portal anomalies (failed auth bursts, validation errors, malicious upload attempts)
  • No IR runbook (rb-00N-forms-*.md) for forms-portal incidents
  • No DR procedure for forms-portal in drp.md
  • No red/blue drill in tabletop-playbooks.md

Compliance framing:

  • PCI DSS: not applicable (no cardholder data flows through forms portal)
  • NIST 800-53 / 800-171: applicable as best-practice — control families AC (Access Control), AU (Audit & Accountability), SC (System & Communications Protection), SI (System & Information Integrity). Audit log + KMS encryption partially satisfy AU/SC; AC + SI need observability work.
  • MITRE ATT&CK: not a compliance framework, but rest of GPUS infrastructure maps detected events to MITRE techniques. Forms portal events should reach the same taxonomy.
  • OWASP ASVS: directly relevant to the SPA + API surface; needs explicit verification pass.
  • CIS: applies to host hardening (not Cloud Run services directly); covered for forms portal's underlying platform.
  • GPUS internal IRP/DRP framework: forms portal needs runbook + DRP + drill coverage matching what the rest of infrastructure has.

Sub-items (provisional scope — refine when work starts):

a. Audit log → Cloud Logging structured events. Forms-backend audit_log table inserts should also emit structured Cloud Logging entries with proper severity. Pipeline: forms-backend → Cloud Logging → log sink → CEDAR (Elastic) for indexing.

b. Wazuh rule additions for forms-portal events. New rule IDs in the 100020+ range (memory entry on Wazuh ruleset). Cover auth_failure (HIGH severity), attachment_rejected (MEDIUM — potential abuse), submission_created (LOW — informational), attachment_uploaded (LOW). Emit MITRE technique tags where applicable.

c. Prometheus metrics from forms-backend. Standard four (request count, latency, error rate, in-flight) plus domain-specific (submissions_per_minute, auth_failure_rate, upload_size_p99, etc.). Scrape via MAPLE.

d. New Forms tab on soc.greenpeace.us. Tab structure: submission volume, auth-failure rate, attachment activity, slowest queries, recent rejections, threat hunting view (filter audit_log by anomaly patterns).

e. IR runbook rb-006-forms-portal-incident.md. Cover scenarios: compromised submitter account, mass-upload abuse, infected attachment in GCS (forward-look at ClamAV scenario), Cloud SQL unavailability, KMS key rotation, DEK compromise.

f. DR procedure for forms-portal in drp.md. Cover Cloud SQL point-in-time recovery, GCS bucket recovery, Cloud Run rollback pattern, Okta tenant outage fallback.

g. Red/blue drill in tabletop-playbooks.md + blue-team-drills.md. Scenario: malicious attachment uploaded by compromised submitter. Validate detection pipeline end-to-end.

h. OWASP ASVS verification pass. Walk the standard against the forms portal surface; document gaps.

Estimated scope: 3-5 sessions if done as a focused workstream. Could be done incrementally if items a-d are prioritized first (functional observability) and e-h follow (process + verification).

Dependencies:

  • Phase 2.5(b)/(c)/(d)/(e) ideally complete first (gives stable surface to instrument)
  • Wazuh rule slot range coordination (memory entry ranges)
  • Cooperation with T4 SOC Tickets workstream (forms events should auto-ticket)

T1.7 — Forms portal frontend: client-side validation must block submit, not just attachment

STATUS: FILED 2026-05-08. COMPLETED 2026-05-14 (commit f175810 shipped 2026-05-12; browser-verified end-to-end 2026-05-14: oversized blocks submit, wrong-MIME blocks submit, clear re-enables, positive case sub 9ac940c8).

Severity: Medium (silent data loss, recipient-visible).

Scope: Frontend only (forms-frontend SPA).

Bug: When client-side validation rejects an attachment (oversized, wrong MIME, or other triggers), the SPA hides the attachment but does not disable the submit button. Submission proceeds to the backend without the attachment. Recipient team sees a complete-looking submission with no attachment and assumes it was intentional.

Known instances (both cleaned up):

  • d20d2ac8-c19d-48df-a0a4-4f833a750e9b — 2026-05-08 17:15 UTC, csv test, cleaned in β step 8 (audit_log id=10).
  • e93efbc3-4c6e-4b7a-adc9-169d4aedec70 — 2026-05-08 18:09 UTC, docx test post-env-var-removal, cleaned in β closeout (audit_log id=12).

Probable triggers (uncharacterized):

  • Server-side rejection signal (oversized, wrong MIME)
  • Stale /api/config cache showing the old MIME allowlist (frontend may have cached the pre-cleanup 4-MIME list with text/csv and without docx/xlsx)
  • Possibly other (see closeout doc architecture/forms-phase2.5b-cleanup-closeout.md)

Fix shape (TBD by frontend session):

  • Submit button should be disabled while any attachment field has a validation error
  • OR submit handler should check for unresolved attachment errors before POSTing to /submit
  • OR both

Relationship to other T-priorities:

  • Filed below T1.6 (forms portal SOC/observability integration)
  • Independent of α (ClamAV) and 2.5(c)/(d)/(e) tracks
  • Should be addressed before any meaningful production user testing — silent attachment drop produces forensically-confusing audit trails (submission_created with no attachment_uploaded) and recipient-team confusion

Estimated: 1 session.

T1.8 — Forms documentation set (mkdocs)

STATUS: Planned — depends on the security assessment. The assessment (threat model + ASVS) is the upstream artifact several of these records cite; sequence the assessment first, then the derived docs. All new/updated docs live in the mkdocs portal.

Assessment (do first):

  • Threat model for the forms portal surface (SPA + API + routing worker + Cloud SQL + GCS + HappyFox path).
  • OWASP ASVS L2 assessment — walk the standard against the live surface; document gaps.
  • PCI-DSS applicability assessment — expected outcome out of scope (no cardholder data flows through forms); record the determination so it's on file, not assumed.
  • Control mapping: OWASP → NIST 800-53 → MITRE ATT&CK, matching the taxonomy the rest of the estate already uses.

Records / runbooks (derive from the assessment):

  • Forms IR runbook (rb-006-forms-portal-incident.md per T1.6) + DRP entry in drp.md with Cloud Run + Cloud SQL RTO/RPO.
  • Security finding record — the broken-access-control finding, its remediation, and the no-exploitation conclusion (empty audit_log + empty users table, independently corroborated).
  • Governance finding — the API contract documented the Phase-1 decrypt/list/submit endpoints as removed while the code kept serving them; nothing checks that the contract's claims match the code. Record the finding and propose a contract-vs-code conformance check.
  • 2.5(d) as-built — document the subject-template audit-persist behavior as shipped.
  • Regenerate the field-exposure matrix (against the authoritative store — see the DB↔repo drift item; regenerating against the repo is invalid if the repo isn't authoritative).
  • Update mitre-attack.md, threat-vectors.md, pentest-schedule.md, calendar.md.

T2 — Meraki

Meraki cleanup — Sedita site + Meraki SSO

  • Goal: Close out the Meraki P2 follow-ups identified after the P1 inventory (org 395909, 5 networks, 32 devices — completed 2026-04).
  • Gating: T1 forms tier complete.
  • Scope:
    • Sedita site misconfiguration — resolve (specifics to be confirmed at start-of-work)
    • Meraki SSO — currently broken, wire to Okta Production
  • Deliverables: Fixes verified end-to-end (Okta login → Meraki dashboard for an admin test user); Sedita site returns to expected operational state.

Meraki integration — status/SOC/MkDocs coverage

  • Goal: Fold Meraki into the same documentation and monitoring posture as the rest of the estate — Meraki currently lacks matching coverage.
  • Gating: Meraki cleanup above complete.
  • Deliverables:
    • Status site: Meraki org card (device count, online/offline, firmware currency)
    • SOC site: Meraki alerts surfaced (security events, config changes, WAN uplink loss)
    • MkDocs: new architecture/meraki-network.md describing org, networks, devices, admin model, SSO posture
    • iar.md entries for Meraki org + each site
    • Syslog from Meraki → WIND (so Wazuh indexes Meraki events into CEDAR)
    • IR runbook: rb-006-meraki-compromise.md (admin account takeover, rogue config push, AP impersonation)
    • DR procedure: Meraki config backup/restore procedure added to drp.md (Meraki backs up config in-cloud, but document how to roll back + how to replace a bricked device)
    • Red/blue drill: tabletop "Meraki admin credentials leaked" in tabletop-playbooks.md; blue-team detection drill for "unexpected config change outside change window" in blue-team-drills.md

T3 — WDC foundation

SKY portal-backup cron died (~2026-04-17) — ADJACENT (non-forms)

  • Status: Filed. Not a forms item, but an open backup-coverage gap worth surfacing alongside the WDC work.
  • What: The gpus-portal-backup.sh cron on SKY (nightly 02:30 → GCS, shipped 2026-03; see Completed) stopped running ~2026-04-17. portals/ snapshots stop at that date. Server backups are current — only the portal-content snapshot stream is affected.
  • Do: Root-cause why the cron stopped, restore it, and backfill the snapshot gap (≈Apr 17 → now). Note this compounds the T5-EXPANDED P0 finding that soc-site + forms-frontend were already unchecked by portal-backup coverage — restoring the cron should also extend coverage to those.

ESXi inventory & cleanup

  • Goal: Bring the ESXi hypervisor — currently undocumented — under the same documentation, monitoring, and IR posture as everything else.
  • Gating: T1 + T2 complete.
  • Deliverables:
    • Inventory: ESXi version, licensing, VMs hosted, networking, hardware health, management plane exposure
    • Confirm or sever ESXi ↔ NAS coupling (decision: is the NAS a datastore, a backup target, or both?)
    • Cleanup: disable unused accounts, rotate admin credentials, enable syslog → WIND
    • Reconfigure: NTP, DNS, timezone, email alerts → gpus-it-security@greenpeace.org
    • Monitoring: Prometheus scrape via vmware_exporter, Grafana dashboard, Wazuh agent on guest VMs where feasible
    • Status site: ESXi card (version, uptime, VM count, datastore usage)
    • SOC site: ESXi in asset coverage map
    • MkDocs: architecture/wdc-hypervisor.md
    • wdc-hostregistry.csv entry; iar.md entry
    • IR runbook: rb-007-esxi-compromise.md (hypervisor takeover, guest escape, management plane breach)
    • DR procedure: ESXi host failure recovery in drp.md
    • Red/blue drill: tabletop "ESXi vCenter creds leaked" in tabletop-playbooks.md; blue-team detection drill "unexpected VM clone / snapshot export" in blue-team-drills.md
  • Known risk: ESXi 6.7 is already flagged as EOL in tracker.md (VLN-004). Inventory may surface the need to accelerate hypervisor replacement — if so, that becomes its own T-tier item.

Synology NAS inventory & cleanup

  • Goal: Same as ESXi above, for the Synology NAS.
  • Gating: ESXi inventory complete (likely coupled — NAS may be serving as an ESXi datastore, which affects cleanup sequencing).
  • Deliverables:
    • Inventory: model, firmware, volumes, shares, users, backup targets, relationship to ESXi
    • Cleanup: disable unused accounts, rotate admin credentials, enable SNMP + syslog → WIND
    • Reconfigure: NTP, DNS, timezone, email alerts
    • Monitoring: Prometheus scrape via SNMP, Grafana dashboard
    • Status site: Synology card (volume health, SMART, firmware)
    • SOC site: in asset coverage map
    • MkDocs: architecture/wdc-nas.md
    • wdc-hostregistry.csv entry; iar.md entry (classification, owner, retention, criticality)
    • IR runbook: rb-008-nas-compromise.md (ransomware on shares, credential theft, firmware tampering)
    • DR procedure: NAS failure + volume rebuild in drp.md
    • Red/blue drill: tabletop "Synology admin portal exposed" in tabletop-playbooks.md; blue-team detection drill "mass file encryption on shares" in blue-team-drills.md

GCP terraform tree has no VCS (supersedes "WDC VPN/route TF state drift")

  • Goal: Initialize version control on ~/terraform/gpus-infra/terraform/ (currently untracked on rchhetry's Mac, single point of failure) so terraform changes can be reviewed, rolled back, and "what's the source of truth" has an answer.
  • Discovered: 2026-05-21, during α ClamAV Commit 2 close-out (A2). git -C ~/terraform/gpus-infra/terraform status returned fatal: not a git repository, and exhaustive .git search across /Users/rchhetry confirmed no repo contains these .tf files. The previously-filed "WDC VPN/route Terraform state drift" item assumed a remote repo existed to drift from — the framing was wrong; the no-VCS problem is the parent.
  • Acute risk mitigation (in effect 2026-05-21): ~/Downloads/terraform-snapshots/2026-05-21/ holds md5-verified copies of all 11 .tf files (no tfvars/tfstate/tfplan copied — those need secrets review first). Local-disk redundancy only; not version control.
  • Inherited drift (still real, blocked until VCS exists): terraform plan against live state shows google_compute_vpn_tunnel.wdc_tunnel local_traffic_selector forcing replacement; google_compute_route.onprem_mgmt + google_compute_route.onprem_prod must be replaced as dependents; google_compute_instance.{cedar,maple,openvas} in-place updates. Replacing tunnel + routes would tear down the WDC↔GCP site-to-site VPN (SKY/RAIN DNS-DHCP, SUN Prometheus, WIND ELK). Interim mitigation: all clamav-worker Terraform applied -target-scoped so the drift is never actioned.
  • Deliverables for the dedicated session:
    • Secrets review of terraform.tfvars + terraform.tfstate* + tfplan: enumerate values, identify what must NOT enter VCS.
    • Design .gitignore: minimum terraform.tfvars, *.tfstate*, tfplan, .terraform/, plus anything from secrets review.
    • git init in ~/terraform/gpus-infra/terraform/; first commit of the 11 .tf files (and any safe-to-commit ancillary files).
    • Decide remote: Cloud Source Repositories (matches existing convention for gpus-infra-portals) vs private GitHub.
    • Push initial commit; document the remote in the worker README.
    • With VCS in place: drift triage — root-cause the VPN tunnel local_traffic_selector mismatch, decide reconcile direction (update Terraform to match live, OR plan a maintenance-window apply that recreates tunnel/routes), if recreation: scheduled change window with WDC-connectivity-loss comms.
    • Remove the -target workaround note from the clamav-worker README once unscoped apply is safe.
  • Gating: independent of WDC inventory work; should run before any further unscoped terraform apply. Best-suited to a dedicated session — touching secrets + remote setup + drift reconcile + maintenance window planning needs full attention.

T4 — Queued

SOC Ticketing tab

  • Goal: Replace ad-hoc alert triage with tracked tickets on soc.greenpeace.us.
  • Gating: T3 complete (stable asset inventory before we wire ticketing to it).
  • Sources: Wazuh (level ≥ 10), Prometheus alertmanager, AIDE change alerts, Fail2ban bans, OpenVAS critical/high.
  • Dedup: 5 min window on (source, rule_id, host).
  • SLA: Critical = 15 min ack / 4 hr resolve · High = 1 hr ack / 24 hr resolve · breach → Slack #soc-alerts + email.
  • Asset docs required: iar.md update; IR runbook rb-009-soc-ticketing-outage.md; blue-team drill for "silent alert drop" (ingestion pipeline broken but tickets still showing green).

Vendor Access Portal (replace SFTP)

  • Goal: Zero-trust replacement for the current SFTP vendor drop.
  • Gating: SOC Ticketing in place (so vendor-portal anomalies ticket correctly from day one).
  • Controls: Vendor IP whitelist via Cloud Armor, signed expiring URLs (max 72h), full audit trail, per-vendor bucket prefixes, ClamAV scan before internal consumption.
  • Asset docs required: iar.md; IR runbook rb-010-vendor-portal-abuse.md (stolen signed URL, vendor account compromise); blue-team drill for "vendor credential used from unexpected geography".

T5 — Backlog

Status site automation — Phase A

  • Goal: Eliminate hardcoded values in status-site/index.html.
  • Scope: Cloud Run service list via gcloud run services list at render time; server count via servers.py; cost block labelled "last updated YYYY-MM-DD" (still manual this phase, but honest about staleness).
  • Estimated effort: 2h.

MySQL decommission

  • Goal: Retire the legacy MySQL instance. Remaining dependencies to be confirmed during inventory.
  • Gating: Legacy migration path confirmed (see Forms Phase 1.5 outcome — likely overlap).
  • Asset docs required: iar.md removal; drp.md update to drop MySQL recovery procedure; final backup captured and sealed in Coldline GCS with 7yr retention lock before shutdown.
  • 2026-06-10: legacy in_formfeed MySQL found on PUBLIC IP 34.171.123.238 — ownership unconfirmed. Decommission precondition: drift check via from_mysql.py --dry-run; authorized-networks check needed.

Status site automation — Phase B

  • BigQuery billing export, live current-month spend, trend, forecast.
  • Requires GPI budget approval for BigQuery storage + query cost (est. <$5/mo).

Status site automation — Phase C

  • Per-service cost attribution, budget alerts.
  • Depends on Phase B (BigQuery export) landing.

T5-EXPANDED — Portal static-debt retirement

  • Filed: 2026-06-10, per read-only audit of both portals (status + SOC).
  • Sequencing: forms 2.5(c) Gate 5 has now landed (go-live 2026-07-23), so this is no longer gated behind it. P0 truth-fixes may interleave with the forms cutover/hardening workstreams; the remainder queues behind them.
  • Phases:
    • P0 — truth-fixes:
      • SOC posture fail-open bug: unreachable host renders green "Compliant"; hardcoded auditd/SELinux/firewall columns.
      • Stale-wrong Exec risk register — DRP/IRP marked "not documented" but both exist.
      • "All 8 services" undercount.
      • Reports last_generated always None.
      • Portal-backup coverage gap: soc-site + forms-frontend unchecked.
    • P1 — wire already-served data: server cards from /api/status; discarded /api/carbon; /api/reports fields; Governance link-out.
    • P2 — author missing canonical sources: defense-in-depth.md, threat-model.md, risk-register.yaml (status & SOC registers currently DISAGREE), compliance-scores YAML — then build-time render runbooks/redblue/compliance (fixes SOC missing rb-006/007).
    • P3 — new collectors: VPN, DNS serial, Prometheus range charts, posture score, FLEET→inventory, Cloud Run table from inventory (overlaps forms Gate 5).
    • P4: BigQuery billing (= old Status-site Phase B, GPI-gated), control matrix, git-log audit trail.
  • Standing rule (recorded 2026-06-10): all portal content must be LIVE or DERIVED — never hardcoded. The coverage standard is to be extended to require that elements RENDER FROM sources, not merely that endpoints exist — the audit found backends over-serving and frontends discarding (e.g. /api/carbon fetched then thrown away).

Cross-cutting / lessons learned

Cloud Run env vars + Cloud Build deploys

Cloud Run env vars set out-of-band via gcloud run services update --update-env-vars are wiped on every Cloud Build deploy IF the cloudbuild.yaml uses --set-env-vars (destructive replace) instead of --update-env-vars (merge). Discovered 2026-04-27 in forms-backend/cloudbuild.yaml — fixed in commit 1ca4461. Other three backends (status, soc, security) don't have this bug because their cloudbuild.yaml files don't pass any env-var flag.

Lesson: any new backend cloudbuild.yaml should either omit env-var flags entirely (preserve) or use --update-env-vars (merge). Never --set-env-vars unless the deploy is intentionally the source of truth for ALL env vars.

Cloud SQL access for developer-side diagnostics

gpus-forms-db is private-IP only (10.34.0.3). Reaching it requires presence inside the gpus-infra VPC. Tested 2026-04-28:

  • Laptop direct: blocked (no VPN to forms-db's service-peering range)
  • Cloud Shell + cloud-sql-python-connector: blocked (timeout to private IP from Cloud Shell's managed network)
  • Cloud Shell + cloud-sql-proxy --private-ip: blocked (proxy bound locally fine but the dial to 10.34.0.3:3307 timed out)

Workable paths for developer-side diagnostic queries:

  • SSH into MAPLE/OAK/CEDAR (all in gpus-infra VPC), run script there. Caveat: that VM's service account principal needs Postgres-side grants (CLOUD_IAM_USER + SELECT).
  • Add VPC peering between Cloud Shell's project network and gpus-infra (administrative work, deferred).

Lesson: Phase 2.5 implementation work needs MAPLE-based or peered DB access established as a prerequisite. inspect_actions.py (forms-backend/migrate/, committed 9eecc35) is ready to run from any in-VPC environment.

Memory entries describing "current bugs" age fast

Diagnostic memos written during one session describe state at that moment, not current state. The 2026-04-22 count-drift memo described a real bug in auth.py — fixed one day later in commit dd810e0 — but the memo persisted in memory and led to a Phase 2.5(a) design draft that proposed re-fixing the already-fixed bug. Code surfaced the staleness by reading current auth.py before any edits.

Lesson: Read-first discipline includes reading current code, not just current memory. When a memory entry describes a bug, verify the bug still exists by reading the affected module's current state (and grep recent commits for likely fix language).

Design doc accuracy under read-first discipline (2026-04-30)

Three design-doc bugs were caught by Code's "STOP and tell me if anything doesn't fit" gate before Commit 3 shipped:

  • _audit helper signature mismatch with actual AuditLog model
  • Invented audit_action enum value not in schema
  • Assumed SQLAlchemy relationship that wasn't declared

All three would have crashed the handler at runtime (or worse, the third would have silently produced submissions with NO field rows).

Lesson: design docs that "mirror Phase 1 patterns" must be drafted from current reads of those patterns, not from memory of earlier reads. The three failures had a common root: the doc described what was remembered of Phase 1's behavior, not what Phase 1 actually does at the line number the design doc claims to mirror. Verifying the source pattern at draft time would have caught all three.

Read-first discipline doesn't end at "read the file once during investigation." It applies again at every implementation moment that references that file's content.

MAPLE access for ad-hoc DB queries (2026-04-30)

Established and verified working pattern for ad-hoc Cloud SQL inspection from MAPLE (Phase 2.5 implementation work depends on this for diagnostic queries):

  • SSH user: cloudadmin (NOT monitadmin, which is SUN/WIND only).
  • MAPLE has cloud-sql-proxy v2 + psql pre-installed.
  • MAPLE does NOT have python3.11 or git — Python connector path requires sudo dnf install.
  • Postgres user maple-agent@gpus-infra.iam exists with SELECT grants on submissions, submission_fields, audit_log (no GRANT needed).

Standard one-liner:

ssh cloudadmin@maple "cloud-sql-proxy --auto-iam-authn --private-ip \
  gpus-infra:us-central1:gpus-forms-db &" && sleep 5 && \
psql "host=127.0.0.1 port=5432 dbname=gpus_forms user=maple-agent@gpus-infra.iam sslmode=disable" \
  -c "<query>"

inspect_actions.py at forms-backend/migrate/ can run from MAPLE only if python3.11 + git are installed first. Defer Python install until there's a real need beyond what proxy + psql handles.

Hypothesis falsification via empirical data (2026-05-07)

α phase pulldown regression demonstrated a clean hypothesis-disconfirmation chain:

  1. Symptom: yes/no pulldowns failing "Could not load options"
  2. Initial hypothesis: missing pulldown rows in DB
  3. Hypothesis falsified: DB query showed all 7 failing pulldown_names exist in pulldowns table with proper ["Yes", "No"] values
  4. Pattern in failing data: all 7 names contained /
  5. New hypothesis: Flask string converter doesn't match /; URL decoding splits the path
  6. Fix: change <name> to <path:name> in route decorator
  7. Verified: dropdown opens with Yes/No options

Lesson: when a fix idea seems obvious ("add the missing pulldown rows"), check the data first. The obvious fix shipped without the falsification step would have been a no-op (rows already exist) and the bug would persist with diagnostic time wasted plus user trust diminished.

This pattern is reusable: when an outage looks like "data is missing," verify by query before shipping the inverse ("add the data"). The reverse pattern — when data exists but isn't being read — points at a different layer (routing, auth, encoding, serialization).

Operational learnings — 2026-06-10

  • L2TP stale tunnel, 3rd occurrence: tunnel reports "up" while the path is dead. Health checks must probe internal IPs, not tunnel state.
  • FortiClient utun6 route confound: FortiClient's interface can shadow routes and confuse VPN path diagnosis — rule it out before blaming the site-to-site tunnel.
  • IAP 4003 = host-level: the IAP-to-MAPLE 4003 backend-fail is host-level, not IAP edge. Promotes the T3 item to IAP-as-break-glass framing (see Summary).
  • Legacy in_formfeed MySQL on PUBLIC IP 34.171.123.238: ownership unconfirmed. Decommission item filed under T5 MySQL decom with drift-check precondition (from_mysql.py --dry-run); needs authorized-networks check.

Completed

Initiative Completed Notes
Forms portal LIVE in production 2026-07-23 Go-live submission 28bd1ebf; routing worker f9a114b on MAPLE, override OFF; all 6 ingest addresses verified delivering; HappyFox tickets opening.
Broken-access-control remediation 2026-07-23 Phase-1 decrypt/list/submit removed; require_role fail-closed; resolve_user least-privilege; IDOR ownership checks on Phase-2. No evidence of exploitation (empty audit_log + empty users table, independently corroborated). Finding record → T1.8.
Forms 2.5(c) routing worker — Gates 3/4/5 2026-07-23 finalize_submission wired; MAPLE-resident Pub/Sub worker via localhost:25 Postfix; migrations 011 + 012 live. Residual propagation verification → Portal-propagation workstream.
Forms 2.5(d) — subject-template audit-persist + GAP-1 renderer fix + template reconciliation 2026-07-23 Audit-persist gate resolved; <%= Grant Funded => GAP-1 renderer fix landed; template reconciliation done.
Forms Portal Phase 2.5(b) — attachment upload wire-up + cleanup 2026-05-08 β closed with verification gap acknowledged (T1.7). Commits e17dacb (handler) + 28964c0 (cleanup). Migration 002 documents submission_deleted enum addition. See architecture/forms-phase2.5b-cleanup-closeout.md.
Okta Production cutover 2026-04-23 Production tenant live; group-based assignment; Preview kept as dev fallback
forms.greenpeace.us DNS + TLS 2026-04-21 CNAME → ghs.googlehosted.com; managed cert issued
Forms Portal Phase 1 (backend) 2026-04-20 Cloud SQL PG15, CMEK, IAM auth, AES-256-GCM envelope, RLS 4 roles
Meraki P1 inventory 2026-04 Org 395909, 5 networks, 32 devices
Portal backup cron on SKY 2026-03 gpus-portal-backup.sh nightly 02:30 → GCS. ⚠ Cron DIED ~2026-04-17 — see the T3 SKY portal-backup item; restore + backfill needed.
Okta Preview SSO across 4 portals 2026-03 OIDC PKCE, shared gpus-okta-auth.js

Change log

Version Date Author Change
v1.17 2026-07-24 R. Chhetry / Claude New T1 compliance item: credential-rotation control never closed out. The quarterly forms-portal-credential-rotation-quarterly control emits records (Q3 2026 evidence committed 61eefff) but is never back-filled — 3 ACTION-NEEDED carried since Q2 (KMS rotation timestamps, Cloud SQL backup verification, HappyFox rotation) + an MFA-enforcement flag; possibly two consecutive quarters with no live console verification. Same class as the contract-vs-code and DB↔repo drift findings — artifacts asserting a state nobody verified. Pre-announcement gate: confirm Okta MFA enforcement before the staff notice — forms authz now rests entirely on Okta identity and the announcement leads with "sign in with Okta".
v1.16 2026-07-23 R. Chhetry / Claude FORMS PORTAL LIVE IN PRODUCTION (go-live 2026-07-23, submission 28bd1ebf). Routing worker f9a114b on MAPLE, override OFF, all 6 ingest addresses verified delivering, HappyFox tickets opening. Phases 2.5(a)–(d) DONE; 2.5(c) Gates 3/4/5, GAP-1 renderer fix, and template reconciliation closed. Broken-access-control remediation COMPLETE — Phase-1 decrypt/list/submit removed, require_role fail-closed, resolve_user least-privilege, IDOR ownership checks on Phase-2; no evidence of exploitation (empty audit_log + empty users table, independently corroborated). Restructured T1 from "finish forms" to cutover + post-go-live hardening. Legacy forms.us.gl3 retires 2026-08-01 (a Saturday — flagged for reconsideration); parallel running until then; LDAP deprovisioning gap argues for cutting over sooner. New/organized T1 workstreams: Cutover (staff notice, decommission, in-flight handling, redirect-or-dark, duplicate-ticket overlap, LDAP gap); Detection pipeline (audit_log→SOC + Cloud Run→Wazuh sinks don't exist; rules 100026–100029 inert by starvation; routing-worker empty SyslogIdentifier) — blocks the drill program; Security hardening (retire auth.py authz; narrow backend-SA over-grants; Cloud Armor = LB+NEG arch change; /metrics + /health/deep exposure; CORS *; per-instance rate limiting; MAC validation); Content/data (<% = Note => in Finance Termination rendering literally; live-DB legacy-tag sweep; DB↔repo drift root cause — HIGH; Insurance checkbox unanswerable-as-No; HR Termination TODO; footer sweep; DSAR/erasure + retention-lock gap); Okta dedicated forms app (requested from Conan 2026-07-23); 2.5(e) HappyFox API (queue names from API, ticket-audit migration, per-queue double-ticket deconfliction, API preferred — DMARC spoofing case); Portal propagation (live/derived, verify pages render). New T1.8 documentation set (threat model, ASVS L2, PCI-DSS out-of-scope determination, OWASP→NIST→MITRE mapping, IR runbook + DRP RTO/RPO, BAC + governance finding records, 2.5(d) as-built, field-exposure regen) — depends on the assessment. T1.9 exercise program (SQLi tabletop 60m + blue-team 90m + red-team 90m) — BLOCKED on detection. Phase 3 HappyFox marked superseded by 2.5(e); T5-EXPANDED un-gated (Gate 5 landed). New adjacent T3 item: SKY portal-backup cron died ~2026-04-17 (portals/ snapshots stopped; server backups current).
v1.15 2026-06-10 R. Chhetry / Claude Forms 2.5(c) implementation underway. Design doc → v0.3 (commits f0b79a8, 9a5281b): §3a wire contract with purged=410 DECIDED, §3b async finalize, §6a actions-schema-as-built, §17 coverage triad, 2.5(e) relabel. Migration 011 APPLIED LIVE (audit_action enum 26→30, commit 4bdfdf9); migration 012 (submission_finalized, 30→31) approved 2026-06-10, applying. Gate 3 in progress per G3.0 decisions (C1 add GET, C2 routing_result finalize marker, C3 full predicate at finalize, C4 α publishes unconditionally, C5 = 012). Gate 4 = routing worker; Gate 5 = propagation (NEW, blocks "2.5(c) done"). 7-day Pub/Sub retention clock starts at Gate 3 push — Gate 4 within the window. New T5-EXPANDED filed: portal static-debt retirement (P0–P4) per 2026-06-10 read-only audit of both portals; P0 truth-fixes may interleave before Gate 4. Standing rule recorded: all portal content LIVE or DERIVED, never hardcoded; coverage standard to require render-from-source. Operational learnings logged: L2TP stale-tunnel 3rd occurrence, FortiClient utun6 confound, IAP 4003 host-level (break-glass promotion), legacy in_formfeed MySQL on public IP (decom precondition filed).
v1.14 2026-06-04 R. Chhetry / Claude Forms 2.5(c) routing pipeline design COMMITTED (v0.2, commit 40f90d5; live at architecture/forms-phase2.5c-design/). 2.5(c) moved from next-up to design-committed / implementation-pending. Locked: transport B-iii (MAPLE-resident Pub/Sub pull worker, localhost:25 Postfix, zero new secret, carries its own systemd + monitoring + IR/DR/drill); reuse forms_app role (RLS USING(TRUE) → GATE-4 state-coverage N/A); signed-URL attachment delivery (inline MIME deferred); interim minimal-plaintext body NOT gated on 2.5(d); lean audit — migration 011_routing_audit_actions.sql adds 4 enum values (submission_routed, submission_route_failed, email_sent, email_failed). Next code: 011 migration + finalize_submission wire-up (pure stub today); one code-time confirm — _status_to_wire at routes_phase2.py:57.
v1.13 2026-05-21 R. Chhetry / Claude α ClamAV close-out housekeeping. Sigrefresh build trigger now filters by gpus-forms-clamav-worker/** (A0) — fixes the double-build-on-every-push issue. Backfill drain complete (A1) — 3 fixture attachments (PNG/DOCX/XLSX, all rchhetry β-phase test files from 2026-05-08) re-fired via synthetic Pub/Sub publish, all clean, GCS bytes unchanged, 3 new audit_scanned_clean rows; attachments table now clean=5, pending=0. Design doc alpha-clamav-worker.md → v1.3 (§6 sweep reframed required, §10 DLQ subscription clarified, new §13a 7-point Cloud SQL access spec for new DB-using services). Commit 3 hardening row added to summary. Four new T3 candidates filed: VPN cold-start packet loss, IAP-to-MAPLE 4003 backend-fail, Cloud Scheduler missed-tick on cold-start, and "GCP terraform tree has no VCS" (the no-VCS finding supersedes the previously-filed "WDC VPN/route Terraform state drift" — drift framing was wrong; no remote ever existed to drift from). A2 deferred per the no-VCS surprise; tonight's mitigation is a local snapshot of the 11 .tf files.
v1.12 2026-05-20 R. Chhetry / Claude α ClamAV Commit 2 COMPLETE. Slack alerts + GCS quarantine tag + /dlq-alert + /sweep-stuck + Cloud Scheduler wiring shipped. EICAR test verified full infected pipeline end-to-end to #us-soc-alerts. Worker rev 00010-gr7. Migrations 009 + 010 (audit_action enum + scanner UPDATE/INSERT WITH CHECK widen). Terraform 6 resources applied (DLQ sub + scheduler SA + 2 scheduler jobs). 8 hardening/hygiene items filed for Commit 3.
v1.11 2026-05-19 R. Chhetry / Claude α ClamAV Commit 1 COMPLETE. Migrations 004-008 shipped (enum + IAM user + grants + RLS + sequence USAGE). Pipeline verified end-to-end on T1.7 fixture (07c3c9e0) and real upload (4a0f5bd5). Worker gpus-forms-clamav-worker live (rev 00002-dwx, 2Gi). α.1 follow-ups: stuck-scanning sweep, DLQ subscription + alert, backfill drain of 3 remaining pending rows.
v1.10 2026-05-14 R. Chhetry T1.7 forms portal frontend silent-attachment-drop bug closed.
v1.9 2026-05-08 R. Chhetry β phase: Phase 2.5(b) attachment upload + 2.5(b.cleanup) MIME/size truth consolidation COMPLETE. Three-layer verification PASS on 999cf0cc-…; 4-source-of-truth divergence collapsed to Config.ATTACHMENT_*; production env vars removed; schema migration 002 documents submission_deleted audit_action enum addition. Two orphan-intent submissions cleaned up (d20d2ac8, e93efbc3). New T1.7 filed: frontend silent-attachment-drop bug surfaced during β verification — SPA submit must be gated on attachment validation state. Verification gap on rev 00043-d9q acknowledged in closeout doc.
v1.8 2026-05-08 R. Chhetry New T1.6 workstream filed: forms portal SOC/observability integration. Gap surfaced during Phase 2.5(b) scoping — forms portal has been operationally invisible to SOC since cutover. 8 sub-items spanning logging/metrics/Wazuh/SOC tab/runbook/DRP/drill/ASVS. Sequenced after Phase 2.5 functional work, before T4 SOC Tickets.
v1.7 2026-05-07 R. Chhetry γ phase: design doc correction shipped (2de57c9) — forms-phase2.5a-design.md now matches shipped code. α phase: pulldown regression for yes/no booleans CLOSED (f31ec5f) — Flask route converter fix. T1.5 sub-section now fully closed (all 4 items a/b/c/d). New cross-cutting lesson on hypothesis falsification via empirical data added.
v1.6 2026-04-30 R. Chhetry Phase 2.5(a) COMPLETE — 3 commits (07cb75c, 10bcd0d, 83f85cb), submissions now persist end-to-end (verified API + DB + audit). γ phase shipped same day: font darkness (f5c79e0+e58dabc), NO_DATA_TYPES filter (6d9231a). T1.5 items b/c/d CLOSED; item a (pulldown regression) remains. Cross-cutting lessons added: design doc accuracy, MAPLE access pattern.
v1.5 2026-04-29 R. Chhetry Phase 2.5(a) Commit 1 SHIPPED (07cb75c, auth_v2 User.username via preferred_username). Commits 2+3 paused at design-complete state. New T1.5 sub-section "Forms Phase 2 UX gaps" added covering pulldown regression (yes/no booleans), COST CENTER duplicate, NOTESDIVIDER label leak, and font contrast.
v1.4 2026-04-28 R. Chhetry FieldRenderer pulldown bug FIXED (commit 5512d8e). Phase 2.5 "Phase 2 backend wire-up" promoted as T1 sub-section after discovery that Phase 2 submission stubs persist nothing. Cross-cutting lesson on Cloud SQL access added.
v1.3 2026-04-27 R. Chhetry Forms Phase 2 frontend cutover complete. Phase 2.1 cleanup queue (5 items) added under T1. FieldRenderer pulldown bug added. Cross-cutting Cloud Run env-var lesson documented.
v1.1 2026-04-24 R. Chhetry Re-sequenced: forms (T1) → Meraki (T2) → WDC foundation (T3). WDC items no longer compete with forms momentum.
v1.0 2026-04-24 R. Chhetry Initial draft — consolidated priorities from memory + recent session notes