Skip to content

VLN-011 — HappyFox dispatch has never run: two forms delivered nothing, and the portal said otherwise

Classification: CONFIDENTIAL — Internal Use Only Document: security/finding-2026-08-06-forms-happyfox-never-dispatched.md · v1.1 · 2026-08-10 · GPUS-IT Status: it-support-request CLOSED end to end (deployed + ticket-verified, see §11). new-employee-notification-contractor-intern still HELD. PART 3 recommendation only.

1. Summary
Finding ID VLN-011
Severity High — silent loss of user-submitted requests
Affected Forms portal; it-support-request, new-employee-notification-contractor-intern
Trigger Deliverability concern raised by Mike Sedita, 2026-08-06
Confirmed 2026-08-06, read-only against live Cloud SQL + repo

routing_worker.py:680-694 matches every happyfox_template action, writes status='deferred', records success=true, and continues. No HappyFox client exists anywhere in the repo — HAPPYFOX_URL is declared in config.py:78 and never read by any code.

0 of 37 submissions have ever received a HappyFox ticket id. 19 sit at happyfox_status='deferred'; none has ever advanced.

Forms with an email action still deliver by email. Two forms had no email action at all, so they delivered nothing — while the UI told every submitter "The relevant team has been notified."

2. The traced submission

d50ba4af-e710-4c58-ac73-e8aa7d948316, submitted 2026-08-06 12:43:28 UTC. Every hop has positive evidence; nothing went dark:

Hop Result
Submission row exists, status=routed
Pub/Sub publish + ack yes — claimed 12:43:28.94, 0.9s after finalize
Worker pickup yes — audit row, actor routing-worker, worker_version f9a114b
Actions fired one: happyfox_template, category 71
Dispatch none — sent=0, failed=0, deferred=1
Ticket none 
submission_routed | routing-worker | success=t |
{"sent": 0, "failed": 0, "deferred": 1, "actions_total": 1}

The system truthfully recorded that it deliberately did nothing, marked it success=true, and set the submission to routed. There was no SMTP or API error to find, because no connection was ever opened.

The pattern this shares with VLN-010

Same spine as the inert Wazuh rules: every check an operator would think to run came back green. Row present, worker ran, status routed, audit success=true. The control did nothing. Absence of an error is not evidence of delivery.

3. Scope — which forms lose what

The split is by action composition, not by form:

Form HF actions Email actions Outcome
it-support-request 1 0 delivered nothing — 2 requests lost
new-employee-notification-contractor-intern 3 0 same shape, 0 submissions — latent
dhcp-request, facilities-support-request 0 1 delivering normally (the controls)
employee-termination-notification 5 3 see correction below — not a loss
contract-extension-notification 3 1 partial — loses 3 of 4 legs, unreviewed

Correction (v1.1, 2026-08-10) — the mixed-form row above was wrong

This table counted actions, and the sentence that followed it — "the mixed forms quietly lose the majority of their routing surface on every submission" — generalised from that count without checking who the surviving legs reach. For employee-termination-notification that generalisation is false.

All five deferred queues' owning teams already receive an email copy on the same submission, four of them on the byte-identical template: 81 and 85 via order 7, 92 via order 0, 100 and 96 via order 4. Across 30 deferred legs over 6 submissions, no team ever missed a notification. Verified against the live database 2026-08-10 and recorded as §5.10 of forms-field-audit.md.

The it-support pattern does not transfer. it-support-request was 0 of 1 owning teams notified; Employee Termination is 5 of 5. Applying the same fix would ship four byte-identical duplicate emails per termination and manufacture the second Finance copy that audit §5.6 exists to resolve.

contract-extension-notification (3 HF + 1 email) has not been given the same per-recipient check. Do not assume either outcome for it — the lesson of this correction is that the action count does not answer the question.

4. Queue 71 was a mis-mapping

it-support-request carried happyfox_category: 71. Per the categories API (confirmed 2026-08-03, pinned in governance/scripts/build_audit.py, published in forms-field-audit.md §4):

Queue Category
45 US - IT Support
71 US - IT Infrastructure

The form named "IT Support Request" was configured to open tickets in the IT Infrastructure queue. Corrected to 45 on 2026-08-06 by decision of the Director of IT — in the same commit as the delivery fix, so the mis-routing could not survive the fix.

Do not infer queue ids from co-routed email addresses

Pairing a HappyFox action with an email action on the same template_id resolves most categories (92↔gpus-people@ confirms four times), but the governance record forbids the method and names this exact case: "that method cannot resolve #85 or #71." Category 71 pairs with no email leg anywhere in the estate. The API-confirmed table is the only source.

5. The fix

Deployed and ticket-verified (v1.1, 2026-08-10)

This section was written as "staged, not deployed". It has since shipped and been confirmed by ticket — see §11. The new-employee-notification-contractor-intern hold in §5.3 still stands.

5.1 Delivery restored for it-support-request

An email_template action to gpus-it-support@greenpeace.org, mirroring dhcp-request's action shape exactly. That address is a proven HappyFox email-ingest: ticket #USITS00357259 auto-opened from it on 2026-06-12 (G4.4) with no HappyFox action dispatched.

The happyfox_template action is retained — it is the eventual 2.5(e) path.

TODO 2.5(e) — double-delivery

When HappyFox API dispatch lands, the email_template action must be removed or this form double-delivers: the API leg and the email-ingest leg both open a ticket. Both legs now target queue 45, so a double would land in one queue rather than two — but it must not ship. Recorded in the YAML and in the commit message. Tracked as the 2.5(e) per-queue deconfliction precondition (priorities T3).

5.2 Equipment field type — scope call

The IT field audit records Equipment ("Laptop/Projector Request") as reaching nobody. That is not only a template omission: the field has never stored a row on any submission, including d50ba4af, which did fill RequestedDates.

As a checkbox it was optional, and an untouched checkbox is omitted by the SPA (FormFill.tsx:147), so no submission_fields row is written. Adding {{ Equipment }} to a template without changing the field would render a literal [Equipment: —] marker on every ticket — worse than omitting it, since "not requested" becomes indistinguishable from "form is broken".

Converted to a required YesNo pulldown (v1 → v2), the identical fix applied to dhcp-request.RemoveFromKace in commit e20fe50. div_lock is preserved verbatim but is inert — div_lock and initial_state are loaded into the DB and never read by the SPA, so there is no conditional-reveal behaviour for the type change to break.

This is a scope addition beyond the literal instruction, flagged for veto: it makes a new required field appear on the form.

5.3 new-employee-notification-contractor-intern — HELD

Not changed this session. Zero submissions all-time, so nothing is being lost today. Proposed addresses, all high-confidence and all proven-delivering, and all matching what its sibling new-employee-notification uses for the same queues:

Queue Category Proposed address
81 US - IT Accounts gpus-it-accounts@greenpeace.org
92 US - People gpus-people@greenpeace.org
96 US - Finance Support gpus-finance-support@greenpeace.org

Awaiting confirmation. One risk at a time.

6. The UI asserted delivery it never verified

Submitted.tsx rendered "Your request has been logged. The relevant team has been notified." as static prose outside every conditional — it could not not appear. routes_phase2.py returned a hardcoded "email_status": "queued" regardless of whether the form had any email action.

For it-support-request the compounding was total: the Ticket row rendered nothing at all when the action was deferred, and the Email row showed "Queued · delivery in progress" for a form with zero email actions. No UI state could distinguish this from success.

Fixed:

Element Before After
Headline "has been notified" (always) "queued for delivery" — or, with no email leg, "it has not been sent to a team"
Email row always rendered rendered only when email.count > 0
Ticket row invisible when deferred "Not created", with "delivered to the team by email" when an email leg exists
Hedge line keyed off happyfoxFailed (now unreachable) re-keyed to emailFailed

Finalize now returns a per-channel routing plan counted from the live actions table, so the response can no longer assert a channel the form does not have.

Plan-honest, not outcome-honest

Finalize returns ~1s before the worker claims the message (measured on d50ba4af: finalize 12:43:28.05, claim 12:43:28.94). It structurally cannot report dispatch outcome. Real outcome needs the SPA to poll a status endpoint, which requires relaxing "submitters cannot view submissions after submitting" (forms-api-contract.md §271) — an authz change, deliberately out of scope. That is the real fix.

The zero-email-leg branch is a permanent guard, not a stopgap. It stays after it becomes unreachable for these two forms: it is the branch that would have surfaced this in a day instead of after two lost requests.

7. Why nothing alerted — and what would

forms_happyfox_total is declared at routes/health.py:25 and never incremented, so the Grafana success-rate panel and the prometheus-alerts.yml failure-ratio rule evaluate an empty series forever and can never fire.

The counter is in the wrong process. It is declared in the Cloud Run backend; the deferral happens at routing_worker.py:683, a systemd service on MAPLE with its own registry and no Prometheus surface at all. Wiring it needs either a textfile-collector export that node_exporter on MAPLE scrapes, or a pushgateway — not a one-line fix. That is why the metric was easy to declare and never wired.

Recommended instead — an age-based alert, feasible today. The signal is already in Postgres and needs no worker change:

SELECT max(now() - submitted_at) FROM submissions
WHERE happyfox_status = 'deferred' AND happyfox_ticket_id IS NULL;

Alert on oldest deferred submission > 24h. Age rather than count: a count alarm is permanently firing from the moment it is installed and gets muted within a week, whereas age goes green the moment 2.5(e) drains the backlog. It would have fired on the first submission.

Not built this session.

8. Acceptance — for the live test, when it is run

A ticket exists in queue 45. Not 250 OK.

SMTP acceptance proves the relay took the message, not that a ticket opened. Nothing short of a ticket number distinguishes success from the failure this finding documents.

RESOLVED — this risk did not materialise (v1.1, 2026-08-10)

The predicted failure did not happen, and the claim behind it was wrong.

The worker sends as gpus-it-security@greenpeace.org and the Gmail relay does rewrite the delivered From to alerts@greenpeace.us — that part was correct. But HappyFox accepted it and opened the ticket. Tickets #USITS00365024 and #USITS00365073 both landed in queue 45 from mail whose delivered From was alerts@greenpeace.us.

The priorities tracker's claim that HappyFox blocks greenpeace.us mail is STALE or was never true. It is corrected at its source in gpus-it-priorities.md (T3 2.5(e) precondition). This matters beyond a stale note: that claim was part of the justification for moving to API dispatch in the first place, and it was carried forward for months without a test that would have taken one email to run. The DMARC facts are unchanged and remain a legitimate, separate argument for API dispatch (greenpeace.us is p=none, greenpeace.org is p=reject, help.greenpeace.org publishes no DMARC — so a forged ticket stays plausible). Do not let this correction be read as "email ingest is fine" — it says the block does not exist, not that the spoofing exposure went away.

The acceptance criterion stands regardless, and for the reason §8 gives: the ticket, not the 250 OK.

9. Verification performed (offline)

  • New template rendered through the real dispatch renderer (gpus-forms-routing-worker/template_render.py) — zero stray markers against the full declared field set; no reference to any undeclared key.
  • Form validated through the real loader schema (yaml_schema.FormSpec).
  • Action list confirmed: one email_template → gpus-it-support@greenpeace.org, one happyfox_template → category 45.
  • dhcp-request and facilities-support-request byte-unchanged.
  • Frontend tsc --noEmit clean; backend compiles.

Not verified: any live send. No submission was fired.

Optional-field markers are pre-existing and estate-wide

Rendering the two real submissions shows [Errors: —] / [RequestedDates: —] markers when those optional fields are left blank. This is the renderer behaving as designed ("a missing field is DECLARED, not dropped") and is not new: the working control dhcp-request produces [MACAddress: —], [WifiMACAddress: —] and [RemoveFromKace: —] in real delivered mail today. Changing it would be an estate-wide decision, not part of this fix.

10. References

11. Outcome — it-support-request CLOSED end to end (2026-08-10)

Deployed and ticket-verified. The acceptance criterion in §8 — a ticket exists in queue 45, not 250 OK — is met twice.

Ticket Queue Proves
#USITS00365024 45 delivery restored, after 37 dead submissions
#USITS00365073 45 v3 — RequestedDates as a real date type, Equipment removed, attachment link resolves

The .us From-rewrite risk did not materialise: the relay rewrote the delivered From to alerts@greenpeace.us and HappyFox accepted it. See the corrected §8 admonition — the tracker's "HappyFox blocks greenpeace.us mail" claim is stale or was never true, and has been corrected at its source.

new-employee-notification-contractor-intern remains HELD (§5.3), untouched. Zero submissions all-time, so nothing is being lost.

11.1 Observability limit — the portal can never learn the ticket number

On the email-ingest path, happyfox_ticket_id stays NULL permanently. This is structural, not a bug and not a gap to be closed:

  • The worker's only channel is SMTP. It gets a 250 OK from the relay, which says the relay accepted the message — nothing more.
  • HappyFox mints the ticket number after ingest, in its own system, and never reports it back down the SMTP path.
  • So there is no database query that can answer "did this submission open a ticket?" for any email-ingest leg. Not submissions, not audit_log, not routing_result. The only evidence lives in HappyFox.

Confirmation of #USITS00365024 and #USITS00365073 came from HappyFox, not from the portal — and that is the only way it can come until 2.5(e) dispatches via API and captures the returned ticket id.

What this breaks

The age-based alert recommended in §7 keys off happyfox_status = 'deferred' AND happyfox_ticket_id IS NULL. On a form whose HappyFox leg is deferred but whose email leg delivered fine, that predicate is permanently true — it fires forever and tells you nothing. It is a valid signal for deferral backlog, not for delivery. Do not let a green/red dashboard built on happyfox_ticket_id be read as delivery confirmation on any email-ingest form.

The attachment added to it-support-request in v3 is not an SMTP attachment. The worker appends a footer of V4 signed GCS URLs (routing_worker.py:472-476). Consequences, all live today:

  • The URL is the credential. Anyone holding the link fetches the file — no login, no group membership, no audit of who fetched it. Possession is authorisation.
  • It sits in plaintext inside a ticket system many people can read. The signature is in the ticket body. Every HappyFox agent with visibility of queue 45, plus anyone the ticket is forwarded to, holds a working capability for its lifetime.
  • Lifetime is 7 days, hard-capped. SIGNED_URL_TTL_S is clamped to 604800 at routing_worker.py:116 and cannot be raised by environment. A ticket open longer than a week outlives its own evidence — the link 404s while the ticket is still being worked.
  • The file itself survives for the 7-year retention. Expiry kills the link, not the object. The attachment stays in GCS under the form's retention_days: 2555 long after every URL to it has died.

So the exposure window and the retention window are set by two independent mechanisms that never reference each other: a 7-day bearer token pointing at a 7-year object. Neither is wrong on its own; the pairing is what deserves the record. Not changed here — raising or shortening the TTL, or moving to an authenticated fetch, is a 2.5-series design decision, not a finding fix.