Travel Approval Workflow¶

Classification: CONFIDENTIAL — Internal Use Only Document: platform/travel-approval-workflow.md · v1.0 · 2026-08-19 · GPUS-IT Services: gpus-forms-backend (Cloud Run) · gpus-forms-routing-worker (MAPLE) Data: Cloud SQL gpus_forms · Topic: gpus-forms-approval-notify

About the Platform Services section¶

Every other part of this portal is organised by host. Architecture pages describe SKY, RAIN, OAK, MAPLE, CEDAR; infrastructure pages describe what runs on each; the host registries enumerate them.

The travel approval workflow does not fit that shape. One request moving from submission to a booking decision touches Cloud Run (the API), Cloud SQL (four tables and their row-level security), Pub/Sub (a topic, a subscription and a dead-letter queue), Cloud KMS (the envelope keys behind every encrypted field), and MAPLE (the systemd worker that renders and sends the mail). No single host page can hold it, and splitting it across five would leave the part that matters — the ordering and the failure modes between components — written down nowhere.

Platform Services is for systems like this: a capability that spans hosts, is described end to end, and is named for what it does rather than where it runs.

What the workflow is¶

A travel request submitted at forms.greenpeace.us is routed to three approvers in order:

Step	Role key	Who	How they are chosen
1	`smt_*`	The SMT member the traveller named	From the form's `SMTApprover` pulldown answer
2	`travel_admin_ops`	Shereyll Woodley	Fixed
3	`travel_final`	Kevin Toruno	Fixed

travel-request-001 is the only form with a chain. An unknown form_id resolves to BLOCKED rather than guessing a chain.

DECIDED 2026-08-25 — step 3 is called \"Final approval\", not \"Finance\"

Recorded so it is not re-litigated. Every surface already agrees, and they agree with each other:

Surface	What it says
Role key	`travel_final`
`approval_dispatch.STEP_DESCRIPTIONS` (and its `gpus-reports` mirrors)	`3: "Final approval"`
`forms/travel-request.yaml` instructions	"…then Admin Ops, then final approval"
`guides/travel-approval-final.md`	"the final approver on every travel request", "You are step 3 of 3"
The other three guides	"step 3", "final approver"

Nothing in this repository ties Kevin Toruno to Finance. He is the final approver; that is the whole of the role. The only occurrence of "Finance" anywhere near this form is a commented-out BLOCKED pulldown note at forms/travel-request.yaml:106, which concerns pulldown values pending from Finance and has nothing to do with step 3.

Where the confusion came from. The form's own instructions column used to read "…routes it to the SMT member you name and then to Finance" — served live to every submitter for the entire life of the chain. That text was wrong about the step's name and about the order, and it was corrected on 2026-08-24. See GOV-021 for the correction and its live verification. Anyone who remembers step 3 as "Finance" is remembering the stale text, which no longer exists.

So: do not rename it. A rename would have to move the role key, three copies of STEP_DESCRIPTIONS, the YAML instructions (and therefore the live forms.instructions column, which needs a version bump and a boot-time reload to propagate), and all four staff guides — to replace a name that is accurate with one the repository has no basis for.

The state machine¶

Two independent status vocabularies. The round (submission_approvals.state) answers "where has this request got to"; the step (approval_steps.status) answers "what happened to this one approver".

Round states — six¶

stateDiagram-v2
    [*] --> IN_APPROVAL: chain resolved, ≥1 step PENDING
    [*] --> BLOCKED: chain could not be resolved
    IN_APPROVAL --> APPROVED: last step approved
    IN_APPROVAL --> RETURNED_FOR_REVISION: any approver returns
    IN_APPROVAL --> DECLINED: any approver declines
    IN_APPROVAL --> WITHDRAWN: manual SQL only
    BLOCKED --> WITHDRAWN: manual SQL only
    APPROVED --> [*]
    RETURNED_FOR_REVISION --> [*]
    DECLINED --> [*]
    WITHDRAWN --> [*]

State	Entered when	Terminal
`IN_APPROVAL`	The chain resolved and at least one step is `PENDING`. Restated explicitly on every approve-and-advance rather than assumed unchanged	No
`APPROVED`	An `APPROVE` decision leaves no next pending step. `current_step` set to NULL	Yes
`RETURNED_FOR_REVISION`	Any approver returns. `current_step` NULL	Yes
`DECLINED`	Any approver declines. `current_step` NULL	Yes
`WITHDRAWN`	No code path sets this. It has been in the CHECK constraint since migration 014; migration 021 is the first and only thing that has ever written it, by hand	Yes
`BLOCKED`	Any of the seven resolver failures below. `current_step` is always NULL regardless of what the steps say	No code path out

BLOCKED is still a record. Any steps the resolver managed to resolve before failing are inserted alongside it.

Step statuses — eight¶

Status	Set when
`PENDING`	Default at insert
`DISPATCHED`	The step's snapshot is written and its notification published. The only status from which a decision is accepted
`APPROVED` / `RETURNED` / `DECLINED`	The assigned approver decided
`SKIPPED_SELF`	The step resolved to the submitter's own address (case-folded compare)
`SKIPPED_DUPLICATE`	The same person holds a later step. The earlier step is skipped and the later one stands
`VOIDED`	A step ahead of a returned or declined step, so the rows say why they never ran

Skips are computed in the resolver before any row is written, in a fixed order: SKIPPED_SELF first, because it is the authorisation-relevant reason and must not be overwritten by the duplicate rule. Only still-PENDING steps are downgraded by the duplicate pass.

Both skips and both decided statuses are terminal for that step. Only PENDING and DISPATCHED are voidable — a step that already records what happened is never overwritten by a later step's outcome.

Where Kevin appears twice

Migration 014 seeds ktoruno@ under both smt_ktoruno and travel_final. When a traveller names Kevin as their SMT approver, steps 1 and 3 resolve to him; step 1 takes SKIPPED_DUPLICATE and step 3 stands, so the final approver keeps the decision. If Kevin is himself the submitter, both steps take SKIPPED_SELF instead. A chain in which every step resolves to a skip is BLOCKED — no approver remains.

The four tables¶

All four were added by migration 014_approval_workflow.sql, applied 2026-08-18 as maple-agent@gpus-infra.iam and verified by information_schema readback rather than by the apply exiting cleanly.

`approval_role_members`¶

Who may act for a role_key. Carries the two-identity pair (below), display_name, and is_active. UNIQUE (role_key, okta_email) — one person may hold several roles, which is what lets ktoruno@ sit under both smt_ktoruno and travel_final, and swoodley@ under both travel_admin_ops and travel_coordinator.

`approval_role_display_map`¶

Maps a submitted pulldown string to a role_key. See the display-string join below. UNIQUE (pulldown_name, display_value) is load-bearing: one display string resolves to exactly one role, so the join can never be ambiguous. UNIQUE is deliberately not on role_key, because one role may be reachable from several pulldowns.

`submission_approvals`¶

One approval round per (submission_id, version). Holds state, current_step, blocked_reason and outcome_notified_at.

`approval_steps`¶

The ordered steps of one round. Holds the resolved snapshot, status, the four timestamps (dispatched_at, decided_at, notification_published_at, notification_sent_at), decided_by_email and comment.

Row-level security is not the control here

RLS is enabled on all four tables with a single permissive forms_app policy. relforcerowsecurity is f, and the table owner — maple-agent@gpus-infra.iam, which is also the identity the routing worker runs as — bypasses RLS entirely. Authorisation is done in application code by approval_access, and that is the whole of it. This is fact A of the ASVS scope statement.

forms_readonly is deliberately given no policy and no grant on these four tables. They carry approver identities, decision timestamps and free-text comments; exposing them to the reporting role is a disclosure decision that has not been made. With RLS on and no policy, that role sees zero rows.

The two-identity model¶

approval_role_members carries two addresses per member:

okta_email — the token email claim. The authorisation comparison target.
notify_email — the delivery address.

They are seeded identical today. They are separate columns so that a mailbox change can never silently become an authorisation change.

The separation is carried onto every step: resolved_okta_email and resolved_notify_email are snapshotted independently at dispatch, and neither is ever returned in an API response — publishing them would put the estate's approver roster behind a URL any authenticated member of staff can reach.

A third column, decided_by_email, records who actually decided, taken from the authenticated caller and never from the request body or from a copy of resolved_okta_email. Divergence between decided_by_email and resolved_okta_email means someone other than the assigned approver decided the step — an authorisation bug — and it must stay detectable by query. Do not collapse the two columns.

Snapshot at dispatch¶

approval_steps.resolved_* have two write points and then they freeze:

At persistence. approval_persist.insert_round_and_steps writes resolved_display_name, resolved_okta_email and resolved_notify_email for every step in the chain, in the same INSERT that creates the round.
At dispatch. approval_dispatch.write_snapshot_dispatched re-writes those three for the one step being dispatched, from membership as it stands at that moment, guarded by WHERE status = 'PENDING', and flips the status to DISPATCHED.

Nothing rewrites them afterwards. The freeze is per-step, not per-chain. A membership change between submit and dispatch is picked up for a step not yet dispatched; a change after dispatch is not. That is fact D of the ASVS scope statement.

Why membership changes do not revoke¶

Authorisation compares the caller against the snapshot on the step, never against live approval_role_members. Deactivating or removing a member does not revoke a step already dispatched to them.

This is a deliberate trade, not an oversight: an approver who was legitimately asked to decide keeps the ability to decide even if the roster changes underneath them, and the row records who was actually asked. The cost is that off-boarding does not close an open step. Closing one requires acting on the step — void it, or let the chain reach a terminal state.

A populated snapshot means ASSIGNED. It does not mean MAY READ.

Because persistence fills resolved_okta_email for every step, every approver in a chain holds a populated snapshot from the instant the submission is finalized — before their step is reachable, and while the request is still sitting with someone earlier.

What separates assigned from entitled is status. approval_access.resolve_access_core requires status = 'DISPATCHED' as the second of its four conditions, ANDed with the address comparison; the address alone never grants. Without that half, the step 2 and step 3 approvers could read a travel request the moment it was submitted, while step 1 had not yet decided.

This is pinned by test_approval_access.py::TestFourConditions::test_c2_no_dispatched_step, which asserts that a PENDING step carrying the caller's own address is denied. If that test is ever "fixed" by relaxing the status check, this paragraph is why it must not be.

The four conditions¶

All required, for a grant that permits a decision:

token_email present and non-empty
A step at the current version has status DISPATCHED
That step's resolved_okta_email equals token_email, both normalised
The round's state is IN_APPROVAL

Condition 4 is checked last on purpose, so the log distinguishes round_not_in_approval from no_matching_step. The response body is identical either way — one uniform 403 forbidden for every authorisation failure, so the endpoint is not a UUID-enumeration oracle.

A separate, weaker grant lets an approver who already decided a step re-read that submission. It permits reading, never deciding.

The display-string join, and the BLOCKED guard¶

pulldowns stores options as a JSONB array of bare strings. There is no options table, no option id, no key — one string serves as both label and stored value. A submitted pulldown answer is therefore the display string, encrypted into submission_fields.value_encrypted. No column anywhere can carry smt_sraman.

So the only available join from a submitted answer to a role is string equality on a human display name. approval_role_display_map makes that join explicit and correctable in one place, instead of leaving it implicit in application code.

This is a deliberate second-choice design. The correct fix is a real pulldown_options table with a stable option_key, ruled out of that pass because it would touch PulldownSpec, the YAML loader, the YAML format and every existing pulldown in the estate.

The match is exact. No strip(), no casefold(), no normalisation — quietly repairing a near-miss would hide the drift the check exists to surface.

The consequence, and it is not optional¶

A display value with no row in the map must not route. It sets submission_approvals.state = 'BLOCKED' and records the unmatched string verbatim in blocked_reason. Never default to a fallback approver, never drop the step silently.

A rename in forms/pulldowns.yaml that is not mirrored in the map breaks the join, and BLOCKED is how that surfaces as a visible stop rather than a silent misroute.

Seven conditions produce BLOCKED:

#	Condition
1	`form_id` has no chain configured
2	Submission row not found
3	Submission has no `SMTApprover` value in `searchable_values`
4	The display value has no row in `approval_role_display_map`
5	A role has zero active members
6	A role has more than one active member — ambiguous, and the competing addresses are named in `blocked_reason`
7	Every step resolved to a skip — no approver remains

Conditions 5 and 6 are also evaluated again at dispatch time, where the same failure blocks the round and publishes nothing.

The seeded map is byte-exact and was verified as such

The six Travel SMT Approver values were checked against the repo YAML on 2026-08-18: all ASCII, no leading or trailing whitespace, lengths 12/12/16/12/14/12. Note Felicity Vonsuck — one word, no space. An earlier draft carried Felicity Von Suck, which does not match the pulldown and would have resolved to nothing on every submission naming her.

travel_admin_ops and travel_final get no rows in the map on purpose. Steps 2 and 3 are fixed roles, not traveller-selected, so they resolve from approval_role_members directly and never pass through it.

The producer/consumer split¶

Registered assets¶

Every component below is an entry in inventory.yaml under cloud_services. The IDs are listed here so the link is verifiable in both directions — the coverage check confirms this page names each entity it claims to document, and a reader arriving from the inventory can find the component described.

inventory.yaml ID	GCP resource	Registered
`gpus_forms_backend`	Cloud Run `gpus-forms-backend`	publisher
`gpus_forms_approval_notify`	topic `gpus-forms-approval-notify`	2026-08-26
`gpus_forms_approval_notify_sub`	subscription `gpus-forms-approval-notify-sub`	2026-08-26
`gpus_forms_approval_notify_dlq`	topic `gpus-forms-approval-notify-dlq`	2026-08-26
`gpus_forms_approval_notify_dlq_sub`	subscription `gpus-forms-approval-notify-dlq-sub`	2026-08-26
`gpus_forms_routing_worker`	MAPLE `gpus-forms-routing-worker.service`	consumer / sender
`gpus_forms_db`	Cloud SQL `gpus-forms-db`	approval tables + RLS

The four Pub/Sub entries carry monitoring_status: planned — no DLQ-depth alert is wired for the approval topics, unlike the 2.5(c) scan pipeline. A message stranded in gpus_forms_approval_notify_dlq today is visible only in the worker's in-process drain to Slack.

There is deliberately no single "approval workflow" entry: cloud_services has no capability-level shape, so the components are registered as siblings under one comment header, exactly as the 2.5(c) pipeline is. That missing grouping is a tracked schema gap, not an oversight.

The Cloud Run backend publishes a pointer. The MAPLE worker renders and sends.

Cloud Run (gpus-forms-backend)
  │  write snapshot + status=DISPATCHED + dispatched_at   COMMIT
  │  publish pointer → gpus-forms-approval-notify
  │  stamp notification_published_at                      COMMIT (separate)
  ▼
Pub/Sub  gpus-forms-approval-notify
  │      sub: gpus-forms-approval-notify-sub
  │      dlq: gpus-forms-approval-notify-dlq-sub
  ▼
MAPLE (gpus-forms-routing-worker.service, systemd — NOT Cloud Run)
     read the row → render → smtplib localhost:25 → Postfix
     stamp notification_sent_at + write the audit row     COMMIT (one txn)

The payload is a pointer, and nothing else¶

{ "kind": "step", "submission_id": "...", "version": 1, "step_index": 2 }

kind is also set as a Pub/Sub attribute, so it is readable in the console and in a subscription filter without decoding the body. An absent kind defaults to step — load-bearing rather than defensive, so messages already in backoff or the DLQ at deploy time still parse.

The outcome message is the same shape with outcome and decided_step_index. That field is deliberately not named step_index, so neither payload can be parsed as the other even with kind stripped.

Why the body never travels in the message¶

Pub/Sub messages are durable and retained. A rendered body in the message would become stored plaintext travel data with a retention profile nobody has decided — a copy of sensitive text outside the database, with none of its access control, sitting in a backlog. The worker reads everything else from the database.

The omission is stated the same way for outcome messages: no rendered body, no approver comment, no addresses, no field values. A declined-travel comment in a message backlog is exactly the copy this rule exists to prevent.

Why MAPLE sends and Cloud Run does not¶

Postfix on MAPLE is loopback-only — inet_interfaces=localhost, mynetworks=127.0.0.1/32. A direct send from Cloud Run failed [Errno 111] Connection refused on submission bb575538 at 2026-08-18 16:33:43. The fix was to send from MAPLE, not to open Postfix to the VPC.

The backend module carries no SMTP_HOST, SMTP_PORT or FROM_ADDR for the same reason the override lives only on MAPLE: a dead copy of a control that reads as configured and does nothing is this estate's recurring failure mode.

What the notification contains¶

The renderer decrypts nothing, and that is the important property of the module. StaffName and Destination are searchable: false, sensitive: true, so they never reach searchable_values and rendering them would require a KMS decrypt. On MAPLE that constraint would otherwise evaporate, because a decrypt is one import away — so it is asserted by test_no_decrypt_symbols_reachable, not by prose.

The mail carries the step number, the SMTApprover value, departure and return dates, and the three cost lines itemised — never a bare total, because composition changes the decision and a zero line is information. It carries no Purpose, no Acknowledgement, no traveller name and no destination, and it says so in the body. The approver follows the Okta-gated link for full detail; that link carries no token and is not a bearer credential.

The read key list is fixed, not iterated from searchable_values, so a future searchable field cannot leak into an approval mail merely by being added to the form.

A cost field added after a request was submitted¶

EventRegistrationFee shipped in form version 4 on 2026-08-31. Four travel requests were IN_APPROVAL at that moment — at form versions 1, 2 and 3, two of them already at steps 2 and 3, i.e. with Shereyll and with Kevin. Every remaining mail on those chains renders from the new COST_FIELDS, against searchable_values that cannot contain the key.

A field that did not exist is not a missing value, and must not mark the total partial. _money(None) returns ("(not provided)", None), which flips the renderer's complete flag and prints:

  Registration fee:   (not provided)
  Total (partial):            500.00   [incomplete — one or more cost lines unreadable]

On a request that predates the field that sentence is false three times over: nothing is unreadable, nothing is missing, and the traveller did not mis-enter anything. It would have reached the coordinator, the final approver and the traveller, on real trips.

So the renderers take form_version and consult COST_FIELD_SINCE. A key that is absent from searchable_values on a form version that predates the version introducing it renders (not asked) and leaves the total complete:

  Registration fee:      (not asked)
  Total:                      500.00
  [Lines marked (not asked) were added to the form after this request was
   submitted. The total is complete for the form as it stood.]

The total genuinely is complete — for the form that was filled in.

Why this is not simply key not in sv, which would be simpler and wrong

A bare absence test would also swallow a key dropped by a future bug: the mail would read (not asked) and the total would look sound. This estate's characteristic failure is a control that reads correct while doing nothing, and a renderer that quietly excuses any absence is one.

The suppression is therefore narrowed by both conditions — the key is absent and the submission's form_version is below the version that introduced it. Every other absence keeps the old, loud behaviour:

Case	Renders	Total
v3 submission, v4 field absent	`(not asked)`	complete
v4 submission, v4 field absent — cannot happen legitimately; the field is required	`(not provided)`	partial
`form_version` unknown (`None`)	`(not provided)`	partial
Value present, including `0`	`0.00`	complete, and the zero is kept

form_version=None degrades to the false-incomplete side deliberately. An over-cautious "partial" is recoverable by looking at the request; a wrong Total: is not, because nothing invites a second look.

Pinned by test_approval_notify.TestCostFieldPredatingASubmission, including the non-masking case and byte-equivalence with the backend mirror across all four rows above.

Adding the next cost field

Add the pair to COST_FIELDS in both renderers, and add "<Key>": <form version> to COST_FIELD_SINCE in both. Omitting the second is what produces a wrong (not provided) on every in-flight request. The equivalence suite will catch a COST_FIELDS divergence; it cannot catch a missing COST_FIELD_SINCE entry, because both copies would be wrong identically.

The two notification columns¶

Column	Written by	When	What it proves
`notification_published_at`	The backend, Cloud Run	After a successful Pub/Sub publish, in its own transaction	The backend published the pointer. Nothing more
`notification_sent_at`	The worker, MAPLE	After `smtplib` returns without raising, guarded `AND notification_sent_at IS NULL`	An MTA on MAPLE accepted the message. Nothing more

Neither is delivery. notification_sent_at means the relay took the message. Bounces downstream are not visible to the portal. This is fact E of the ASVS scope statement.

published	sent	Meaning
NULL	NULL	Never published. The dispatch did not complete, or the publish failed
SET	NULL	Published; the worker has not sent. In flight, retrying, or in the DLQ
SET	SET	Handed to the relay
NULL	SET	Should not occur

The durable write always precedes the side effect: a publish failure leaves the row DISPATCHED with both columns NULL — recoverable and visible. The reverse, published but unrecorded, is worse.

outcome_notified_at is the same idea one level up, on the round rather than the step: an MTA accepted the submitter's terminal-outcome email. It is deliberately a separate column rather than a reuse of notification_sent_at, because the step handler guards on that column — writing it for a submitter notification would make the guard fire against a step whose approver has not been mailed at all, and that approver would simply never hear.

A terminal round with outcome_notified_at still NULL is the recovery query: every traveller whose request is finished and who was never told.

Two rounds predate the column

30a286e1 (APPROVED, 2026-08-19 16:11Z) and 6418f160 (RETURNED_FOR_REVISION, 2026-08-19 16:30Z), both submitted by tgarg@greenpeace.org, went terminal before migration 019 added the column. The deploy does not reach back for them; they need a manual backfill.

The recipient override, and the audit row that records a redirect¶

FORMS_APPROVAL_RECIPIENT_OVERRIDE, set in /etc/gpus-forms-routing.env on MAPLE. When set, every approval mail goes to the override address instead of the approver, with [REDIRECTED — NOT SENT TO APPROVER] prefixed to the subject and a banner in the body.

It is read per send, not once at import — deliberately diverging from the routing worker's own override, which is a module constant needing a systemctl restart. This knob exists to make a first live run safe, and "cleared but still redirecting until someone restarts" is the failure it must not have.

Why there is an audit row and not a column¶

On 2026-08-18/19 the override was ACTIVE while four approval steps were dispatched. All four rows showed status = DISPATCHED with notification_sent_at stamped — the database's own account was "these four approvers were notified". None of them was. Two travel requests departing 2026-08-21 sat waiting on approvers who did not know they existed. The only record was a WARNING in the journal, and the journal rotates.

Migration 020 added the approval_notification_sent audit action. The worker writes one row per send, in the same transaction as the stamp, so a stamped notification always has a row saying where it went — the two facts cannot diverge.

{
  "kind": "step",
  "intended": "swoodley@greenpeace.org",
  "delivered_to": "rchhetry@greenpeace.org",
  "redirected": true,
  "version": 1, "step_index": 2, "role_key": "travel_admin_ops"
}

redirected is derived from a comparison of the two addresses beside it, never passed in, so it can never disagree with them.

A column such as redirected_to on approval_steps was rejected for four reasons: it holds only the last send, so the remediation re-send would erase the evidence of the thing it fixed; audit_log is immutable where approval_steps is not (fact G); one mechanism covers both step and outcome mail; and soc-log-shipper derives its allowlist from models.AUDIT_ACTIONS, so redirected: true becomes alertable with no new plumbing.

Reading a submission after it is submitted¶

forms-api-contract.md:271 states that submitters cannot list, view, edit or delete submissions after submitting — fire-and-forget. That rule is load-bearing and unchanged by this workflow.

This workflow introduces exactly two exceptions, both stated in the ASVS scope statement (§4.B) rather than discovered later:

An approver may read a submission on which they hold a DISPATCHED step. State-gated, on the step's status and the round's.
A submitter may read their own submission. Identity-gated only — one comparison of submissions.submitter_email against the token email, with no state condition. A submitter reads their own request in any state.

Exception 2 was originally designed with a RETURNED_FOR_REVISION state gate, which was dropped when the endpoint was built. The gate was protecting the wrong thing: the policy exists to stop submitters browsing other people's submissions and editing their own after the fact, and a read of one's own record does neither. Gating it meant the only person who could not find out where a request had got to was the person who submitted it.

Both are decrypt paths, both are audited, and both return one uniform 403 for every authorisation failure.

These two are the complete set. Any third read path found in the system is a defect, not an undocumented feature.

What is NOT built¶

Stated plainly, because a component that appears wired and does nothing is this system's characteristic failure.

No revise or edit path. There is no endpoint that lets a submitter amend a request. version is 1 everywhere; the revise endpoint that would mint version 2 is designed-but-unbuilt (ASVS §2.4). A returned request is dead — the traveller submits a new one, and the returned-for-revision mail says so explicitly rather than implying an edit that does not exist.

No withdraw endpoint. WITHDRAWN has been a legal round state since migration 014 and no code path has ever set it. Migration 021 set it by hand on four test chains. Four design questions are open and deliberately not guessed at: who may withdraw (submitter, coordinator, or the approver holding the step — each is a different rule in approval_access); from which states (IN_APPROVAL yes, BLOCKED probably, APPROVED almost certainly not); whether it notifies the approver holding the dispatched step; and how it interacts with the undesigned revise.

When the real withdraw endpoint is built it must add an approval_withdrawn audit action. A user-initiated state transition with no durable trace is exactly the gap migration 020 was written to close one level up.

No submitter notification on withdraw. _SENDABLE_OUTCOMES contains APPROVED, RETURNED_FOR_REVISION and DECLINED. WITHDRAWN is absent, so a withdrawn round is acked and dropped — no wording exists for that state and a retry will not invent one. The renderer has no WITHDRAWN branch either.

No reporting (R1–R4). Approval-state reporting reads across submissions and approvers and therefore has a disclosure profile unlike any of the four endpoints in scope. It needs its own scope statement before it is designed.

No ORM models for the four tables. models.py mirrors 001_init only; both stores reach the approval tables by reflection.

How you know it's working¶

These are observable signals, not configuration. Configuration that reads as correct is what this system has repeatedly had while doing nothing.

The precedent: the approval drain call originally sat as the last statement of the worker's main loop, reachable only when the routing subscription delivered a message. It executed zero times across a whole day while the boot log cheerfully reported the subscription as configured. The DLQ drain had the same defect from a bare return inside an exception branch. Both are now hoisted ahead of the work that was hiding them.

1. `approval.drain.heartbeat` in the worker journal¶

journalctl -u gpus-forms-routing-worker -n 200 | grep 'approval.drain.heartbeat'

Emitted at WARNING every 300 seconds (DRAIN_HEARTBEAT_S), on every drain iteration including idle ones, and immediately on the first iteration after a restart. The line names itself:

approval.drain.heartbeat RUNNING iterations=… messages=… errors=… since boot — this line is the proof the drain executes; its absence means it does not

If it is absent: the drain is not running. Not "running and idle" — those two were made distinguishable precisely because confusing them is how the reachability defect survived verification and was reported as passing. Expect approval.dlq_drain.heartbeat alongside it.

2. Both notification columns set, published before sent¶

SELECT step_index, status, notification_published_at, notification_sent_at
  FROM approval_steps
 WHERE submission_id = :id AND version = 1
 ORDER BY step_index;

If sent is NULL while published is set: the backend did its half and the worker did not. The message is in flight, retrying, or in the DLQ — check the heartbeat first, then the DLQ subscription.

If both are NULL on a DISPATCHED step: the publish failed. The row is recoverable and this is the state that says so.

If sent is set and published is NULL: this should not occur. Treat it as a data-integrity question, not a mail question.

3. `audit_log` rows with `redirected = false`¶

SELECT created_at, target_id, details->>'intended', details->>'delivered_to'
  FROM audit_log
 WHERE action = 'approval_notification_sent'
   AND (details->>'redirected')::boolean IS TRUE
 ORDER BY created_at DESC;

If this returns rows: mail went somewhere other than the approver. That is the August incident, queryable.

If there are no approval_notification_sent rows at all for a step whose notification_sent_at is stamped: the two facts have diverged, which the single-transaction write is supposed to make impossible. Investigate the worker, not the mail.

4. `boot approval_recipient_override=OFF` in the journal¶

journalctl -u gpus-forms-routing-worker | grep 'boot approval_recipient_override'

Expected on every start:

boot approval_recipient_override=OFF — LIVE DELIVERY to approvers

If it reads ACTIVE to=…: no approver is receiving anything. Steps will still reach DISPATCHED and notification_sent_at will still be stamped. The rows will look correct and the approvers will not know the requests exist.

If the line is absent entirely: the worker did not reach its boot banner, or the unit is running an older build. Either way the override state is unknown, and unknown is not OFF.

References¶

ASVS L2 Scope — Forms Approval Workflow — facts A–G
Forms Portal — forms.greenpeace.us
Forms API Contract
Forms DB Access & Break-Glass
Migrations 014–021 in forms-backend/schema/

Exercising the v3 funding fields — the one-submission test¶

Why this exists: loaded is not exercised

The three funding fields (FundingSource, GrantName, CostCenter) were added to travel-request-001 on 2026-08-27. The form serves them, the approval renderer prints two of them and withholds the third, and both sides are byte-equivalent.

And count(*) WHERE searchable_values ? 'FundingSource' is zero.

Nothing has ever been submitted through these fields. Every claim above is a claim about code, not about the estate: no value has been stored, no approval mail carrying one has been delivered, and no report has read one. That is M-01 in this estate's own words — loaded ≠ reachable — and the only thing that closes it is one real submission.

One travel request, submitted by Rajesh, exercises all of it. The checks below are what it proves and how to confirm each; a submission with no verification afterwards proves only that the form accepted a POST.

Before you start — who receives mail, and it is not nobody¶

CORRECTED 2026-08-27 — this section previously said to pick Kevin Toruno. That was wrong.

The original advice reasoned that Kevin is already the step-3 approver, so naming him as SMT approver would collapse the humans involved to one. The first half is right and the conclusion is inverted.

smt_ktoruno and travel_final are the same mailbox (ktoruno@greenpeace.org). The duplicate rule in approval_resolver.resolve_chain_core skips the earlier step so the final approver keeps the decision — so step 1 becomes SKIPPED_DUPLICATE and current_step is pending[0].step_index, which is step 2. The chain does not stall at step 1; it opens in Shereyll Woodley's queue and dispatches to her on submission.

Verified 2026-08-27 by running resolve_chain_core against the live approval_role_members roster:

SMTApprover = 'Kevin Toruno'          SMTApprover = 'Marco Simons'
  current_step = 2                      current_step = 1
  1: SKIPPED_DUPLICATE  Kevin           1: PENDING  Marco Simons
  2: PENDING  Shereyll  <- mailed       2: PENDING  Shereyll
  3: PENDING  Kevin                     3: PENDING  Kevin

"Step 1 collapses, so nobody is mailed" is false. Naming Kevin is the worst selection for a test, not the safest: it mails the coordinator immediately and puts a test request into the live queue she works from.

Pick any SMT approver who is NOT Kevin Toruno — Marco Simons, Felicity Vonsuck, Kalina Francis or John Hocevar. Sushma Raman is out sick as of 2026-08-31; she is still in the roster and still resolves, so a request naming her would dispatch and sit undecided in a mailbox nobody is reading. There is no technical basis to prefer one of the remaining four; pick whoever you can brief in thirty seconds, and brief them before submitting. They will receive a mail subjected "[GPUS Travel] Approval required — Step 1 of 3" with a link to a real approval page, and may act on it if they have not been told.

There is no selection that mails nobody. The only path to that is SKIPPED_SELF on every step, which requires the submitter to be the approver; the submitter is not in the roster. Every selection mails exactly one person — the choice is only whom, and whether the request parks somewhere harmless.

Step	Recipient	Fires when
1 — SMT approval	the non-Kevin SMT member you picked	on submission — the only mail the test sends
2 — Admin Ops	Shereyll Woodley (`swoodley@`)	only if step 1 is approved
3 — Final approval	Kevin Toruno (`ktoruno@`)	only if step 2 is approved

Leave it at step 1. Do not approve. Steps 2 and 3 never dispatch and Shereyll is never mailed.

Do NOT use the recipient override for this

FORMS_ROUTING_RECIPIENT_OVERRIDE looks like the safe option and is the more dangerous one here. It is global, not per-submission: while it is set, EVERY routed approval mail estate-wide is redirected, so a real traveller's request submitted during the window would have its approval mail diverted and its approver would simply never hear about it. The body says so — to the wrong person. For a single test submission during working hours the blast radius of the override is larger than the blast radius of the test.

Two people receive mail you did not send, and it is not avoidable

Shereyll receives R2 daily at 12:15 UTC. Even parked at step 1 with someone else, the request appears in her chase-list once it crosses the stall threshold, and she will chase it. R2 already reported stalled=2 on 2026-08-27, so she is actively working that list. Withdraw the request the same day — see below.
Rajesh receives the submitter-outcome mail for his own request, and R5 at 13:40 UTC, which will count the trip.

"Nobody outside the test receives mail" is not fully achievable while the daily reports are running. What is achievable is that exactly one approver is mailed, and that both report recipients are expecting it.

What to submit¶

Field	Value	Why this value
SMT approver	any non-Kevin SMT member you have briefed — `Marco Simons`, `Felicity Vonsuck`, `Kalina Francis` or `John Hocevar`	Naming Kevin collapses step 1 to `SKIPPED_DUPLICATE` and dispatches straight to Shereyll. See the correction above — this row said `Kevin Toruno` until 2026-08-31 and contradicted it
Funding source	`Cost Center`	the branch that populates `CostCenter`
Cost centre	`20100 — INFORMATION TECHNOLOGY`	em-dash format; the submitter's own department
Grant name	(leave blank)	proves the optional field accepts blank
Already booked	`No`	keeps the request out of R3's numerator
Everything else	anything plausible

Then submit a second request differing only in funding source — Grant, a grant name, and cost centre left blank — if you want the grant branch exercised too. It is the branch where a leak would actually happen.

What it proves, and how to check each¶

SUPERSEDED 2026-08-31 — a REAL submission arrived before the exercise ran

c84bfb45 (Madison Carter, 15:46:12Z) is the first genuine end-user travel request since the v3 bump. It closes checks 2, 3 and 5 on real usage, which is stronger evidence than this exercise would have produced. Check 6 it does not close — her ticket was rendered by the old template 6m06s before the fix landed (VLN-050, reopened).

The exercise is no longer needed for 1–5. If it is run at all it is now only for check 6, and the guidance below stands unchanged for that purpose. Do not run it to re-prove what a real user already proved.

Pre-submission status, 2026-08-31 — two of six already closed

Checks 1 and 4 do not need a submission: they are properties of what is served and what is deployed, and both were verified before the exercise. Rajesh's submission is needed for 2, 3, 5 and 6 only.

#	Check	Status
1	Pulldown serves 31 values in Finance's order	✅ PASS — served digest equals the repo's
2	Stored value em-dash byte-exact	⏳ needs the submission
3	`FundingSource` required, dependents optional	⏳ needs the submission
4	Approval mail carries the funding fields, withholds the grant name	✅ PASS — deployed module, digests equal
5	`count(*) ? 'FundingSource'` moves 0 → 1	⏳ needs the submission
6	The HappyFox ticket body carries the funding lines	⏳ needs the submission — new, see `VLN-050`

Check 4, run against the DEPLOYED renderer, not the repo copy:

$ /opt/gpus-forms-routing/venv/bin/python3 verify_deployed_render.py \
    --module /opt/gpus-forms-routing/approval_render.py
DIGEST  9170c007c4565b0d9e1123d52f9866beacecde721798132dbc10009317e507af
RESULT: PASS

The backend's copy prints the same digest, which is the byte-equivalence assertion. Note MAPLE's default python3 is 3.6.8 and cannot parse the module — use the worker's own interpreter, /opt/gpus-forms-routing/venv/bin/python3, which is also the one systemd runs.

Check 6 is new, and the exercise would have been the first to hit it

VLN-050: forms/templates.yaml → travelrequest001 — the body of the ticket this form raises into queue 45 — enumerated a 13-field form and never gained the three v3 funding fields. Fixed and deployed 2026-08-31 (templates.updated_at = 15:52:38Z, body 531 → 640 bytes, all three placeholders present).

It matters here because no travel request has been submitted since the v3 bump — 12 all-time, 0 since 2026-08-27. So this submission would have been the first ticket to carry the defect, and the exercise would have proved the funding fields reach two surfaces while silently missing a third. Verify the ticket body in HappyFox, not the YAML.

1. The Travel Cost Center pulldown serves 31 values in Finance's order.

The count and the ordering both matter: the values came from Finance's Accounts export verbatim, and a re-sorted or de-duplicated list is a third vocabulary nobody recognises (GOV-024).

CORRECTED 2026-08-31 — the command previously here could not pass

This step used to give an unauthenticated curl piped through json.load(sys.stdin)['pulldowns']. It returns 401 {"error": "unauthorized"} — /api/forms/<id> is Okta-gated — and the response has no pulldowns key even when authorised. So the check could only ever fail, and would have failed identically whether or not the pulldown was correct. An acceptance command that cannot pass is not a check (M-13); it teaches the next person to skip the step.

Two ways to do this for real. Both read what is served, not the repo YAML.

(a) In the browser, logged in — open the travel form and the Cost centre pulldown. Count is 31 and the list opens 10000 / 10200 / 10300. This is the only variant that exercises the actual serving path end to end, and it is what Rajesh is doing anyway to submit.

(b) Against the served pulldowns row, which is what the API renders from — runnable without a browser, and it compares rather than eyeballs:

-- as maple-agent; see forms-db-access.md
SELECT count(*) AS n,
       bool_and(v LIKE '%' || U&'\2014' || '%')        AS all_em_dash,
       bool_and(octet_length(v) > char_length(v))      AS all_multibyte,
       md5(string_agg(v, E'\n' ORDER BY ord))          AS served_md5
  FROM pulldowns p,
       LATERAL jsonb_array_elements_text(p.values) WITH ORDINALITY AS t(v, ord)
 WHERE p.name = 'Travel Cost Center';

# The other half — the same digest, computed from Finance's list in the repo.
python3 -c "
import yaml,hashlib
d=yaml.safe_load(open('forms/pulldowns.yaml'))
items = d['pulldowns'] if isinstance(d,dict) else d
v=[p for p in items if p['name']=='Travel Cost Center'][0]['values']
print(len(v), hashlib.md5(chr(10).join(v).encode()).hexdigest())"

Verified 2026-08-31: n=31, all_em_dash=t, all_multibyte=t, served_md5=a376c3ce026e4d1649a4489a109442da — and the repo digest is the same value. Comparing digests beats eyeballing three rows: it asserts the whole list and its order at once, which is the property GOV-024 is about. Order is Finance's, not numeric-by-accident, so do not "verify" it by checking the numbers ascend.

2. The stored value is the em-dash format, byte-exact in searchable_values.

The separator is U+2014 (—), not a hyphen. A client that normalised it would produce a value that matches no Finance record and no other submission, and it would be invisible on a rendered page.

-- as maple-agent; see forms-db-access.md
SELECT searchable_values ->> 'CostCenter' AS cc,
       searchable_values ->> 'CostCenter' LIKE '%' || U&'\2014' || '%' AS has_em_dash,
       octet_length(searchable_values ->> 'CostCenter') AS bytes
  FROM submissions
 WHERE form_id = 'travel-request-001'
 ORDER BY submitted_at DESC LIMIT 2;

has_em_dash must be t. bytes must exceed the character count — a UTF-8 em-dash is three bytes, so byte length > character length is the cheap proof it survived the round trip.

3. FundingSource is enforced required; GrantName and CostCenter accept blank.

This is the shape the form has, and it is deliberate — the platform has no conditional display and the server cannot validate cross-field conditions, so "GrantName is required when FundingSource is Grant" cannot be enforced anywhere. Both dependent fields are optional at the schema level and the descriptions carry the rule.

Check by attempting a submission with funding source unset — it must be rejected — and by the successful submission above, which left GrantName blank. Confirm the recorded consequence too: a request with FundingSource=Grant and GrantName blank is accepted. That is known and documented; verify it is still true rather than assuming it was fixed.

4. The approval mail carries "Funded by" and "Cost centre" and NOT the grant name.

Check the delivered mail in Kevin's inbox — or ask him to forward it — and read the body:

Funded by:         Cost Center
Cost centre:       20100 — INFORMATION TECHNOLOGY
...
Traveller name, destination and grant name are not included in this email.

Do not grep the source to check this. approval_render.py discusses GrantName at length in its comments, and the rendered body itself contains the words "grant name" in the sentence saying it was withheld. A grep reports a leak that is not one. The check that works is rendering with a sentinel value: gpus-forms-routing-worker/verify_deployed_render.py, run against the deployed module.

5. R5 sees the new keys.

R5 does not read searchable_values at all, by design — so the honest form of this check is that R5's trip count increases by one in the next 13:40 UTC edition, and its cost is one row in the volume series that was a test.

The report that will actually read the funding keys is R1, monthly. Its funding section is coverage-first precisely because of this: the first edition after the test will read "1 of N trips recorded a funding source", and the line naming 2026-08-27 is what stops that being read as non-compliance.

-- The direct check, and the one that closes the M-01 gap:
SELECT count(*) FILTER (WHERE searchable_values ? 'FundingSource') AS funding,
       count(*) FILTER (WHERE searchable_values ? 'CostCenter')    AS cost_centre,
       count(*) FILTER (WHERE searchable_values ? 'GrantName')     AS grant_name,
       count(*)                                                    AS total
  FROM submissions WHERE form_id = 'travel-request-001';

funding moving from 0 to 1 is the whole point of the exercise. Until it does, every statement about the funding fields is a statement about code.

Cleaning up — withdraw, do not tell, and do not delete¶

Withdraw the same day. The alternative — leaving it up and telling Shereyll — costs her attention on a request that is not hers and never will be, and it only defers the cleanup. Withdrawing costs nothing and removes the need to interrupt her at all: the request never crosses R2's stall threshold, so it never reaches her chase-list in the first place.

Withdrawing does not destroy the evidence, which is why it is the cheap option. WITHDRAWN is a state on the approval round (submission_approvals.state); the submissions row and its searchable_values are untouched. So after withdrawal:

count(*) WHERE searchable_values ? 'FundingSource' still reads 1 — the M-01 closure holds
R1 still counts the trip, because R1 counts by submitted_at, not by outcome
R5 counts it at 13:40 either way, withdrawn or not

Do not delete the row. A deleted submission leaves the estate exactly where it started — a claim about code with no record behind it.

Document · platform/travel-approval-workflow.md · v1.4 · 2026-08-31 · GPUS-IT · Classification: CONFIDENTIAL — Internal Use Only