Session Handoff — 2026-09-09¶

v1.2 | 2026-09-09 | Classification: CONFIDENTIAL — Internal Use Only

Purpose¶

One session had accumulated five workstreams. This document closes it and routes every open item to the chat that owns it. It is a routing document, not a status summary — each item names where it goes and what the next person needs to know that is not already in a register.

Written 2026-09-09. Register state at close: gap register v1.30.

Commits: 5c0cab6 (GOV-039 applied, GOV-041/042 raised), dae47e0 (GOV-041 resolved, GOV-043 raised), 6f84a75 (GOV-044, WIND enumerated, Webmin closed, this document), and today's final commit adding GOV-045.

1. Closed this session¶

Item	Outcome
`GOV-039` (1) and (2)	Logstash timezone + dead filter block fixed on CEDAR, verified against the index
`GOV-041`	NTP restored on SKY and RAIN, root cause found and it was not what the row assumed
`GOV-040` scope	Answered — Puppet does not manage the nameservers at all
Webmin	Disabled on MAPLE, `reset-failed` on both; 10000 closed on both hosts
`procedure.md` Step 5	Corrected — it taught three separately-recorded defects
`GOV-045`	DHCP now reaches CEDAR from both nameservers — it never had before
`GOV-044`	Raised — our own post-change protocol trips the detection
`GOV-039` (3)	WIND enumerated read-only; framing overturned twice

Not closed: the GCP firewall split. See §5.3.

The one-line versions, because the reasons matter more than the outcomes¶

GOV-039 was a collector-timezone defect that fabricated a 4.5-hour outage. Fixed forward-only. The proof is a document that agrees with its own payload's zone-qualified clock, not one that merely looks current.
GOV-041 was one invalid word: denyall where chrony 4.5 requires deny all. chronyd was enabled and had never started successfully since the hosts were built in February. The row's own proposed remedy would not have worked, and its acceptance test would have failed the working fix.
Webmin on MAPLE was held back deliberately so two changes were not in flight on the SIEM pair at once. Verified after: Wazuh ingestion moving, five agents reporting, SOC portal rendering.

2. Route to: ENTERPRISE ARCHITECTURE¶

2.1 The inventory gap, in full¶

gpusa has zero inventory.yaml entries across 25 VMs, including both DNS servers. Four projects are unenumerated entirely:

gpus-data-youtube
gpusa-staticsites
wordpress-140819
race-to-the-top-web-app

Plus a third billing account. This is the largest single gap in the coverage standard and it is not a documentation problem — the standard's whole premise is that inventory.yaml is the source of truth, and a quarter of the estate is outside it.

2.2 What actually configures SKY and RAIN — unknown, and it governs the two most critical hosts¶

Enumerated on both nameservers 2026-09-09: no puppet package in the rpmdb, no /opt/puppetlabs, /etc/puppetlabs, /etc/puppet, /var/lib/puppet or /opt/puppet, no puppet binary, puppet/puppet-agent/pxp-agent all inactive, no converge cron or timer, and no visible file-distribution path from phoenix.

And yet something wrote byte-identical 244-byte /etc/chrony.conf files a day apart in February — SKY 2026-02-26 10:05:13, RAIN 2026-02-27 11:25:32. A human repeating a build step is the simplest explanation and it is not established.

Two consequences:

The chrony fix cannot be reverted by a converge. That was checked before it was applied.
GOV-040 needs re-framing. d088b4bf is not "managed but never run" on these hosts — the unit is not managed at all, and Puppet is not the route to fixing BIND chroot there.

2.3 No internal NTP server exists¶

MAPLE, CEDAR and SUN all time out on udp/123. The only reachable source from the WDC segment is the public rhel.pool.ntp.org.

So DNS now depends on the VPN tunnel for time, and DNSSEC validity is wall-clock bounded. If the tunnel is down long enough, the nameservers free-run again at ~5 s/day and nothing notices. An internal source is proposed, not built — deliberately, because siting it is an architecture decision (MAPLE is the obvious candidate but it is also the SIEM).

2.4 The reboot-cadence finding, which generalises beyond time¶

Both nameservers are VMware guests with vmtoolsd active, and VMware Tools syncs guest time at power-on. The 121-day drift matches uptime since the 2026-05-11 boot exactly. The February-to-May interval drifted too and was silently corrected by that same reboot.

The estate did not have time discipline that lapsed. It had a reboot cadence frequent enough to conceal that it never had any.

That shape is the transferable part. Anything whose correctness is silently restored at boot looks healthy for exactly as long as reboots stay frequent, and degrades invisibly once they stop. The 121-day uptime that followed the HVAC event is what made a four-month-old absence finally measurable. Worth a deliberate sweep for other examples rather than waiting for the next long uptime to expose them one at a time.

2.5 Also here¶

Puppet-to-Terraform migration, the Rocky migration, yum.us.gl3 down since 2026-07-15, and the consolidation phases.

3. Route to: SOC / FINDINGS¶

3.1 Carried forward, unchanged¶

VLN-053, VLN-054, GOV-038, GOV-039 (3) and (4), GOV-040, GOV-042, VLN-009 sudoers remediation, the NSEC3 salt reverting VLN-014's pinned salt weekly, RAIN's aide.conf monitoring paths that do not exist, the sentinel finding, PORTAL_DIRS two-vs-three, soc-site rendering neither forms entity, rc2 in production three GA releases behind, gpus-soc-alerts@ pre-flight, the 100010 curl exclusion.

3.2 New — `GOV-043`, journald persistence absent on ALL FOUR WDC hosts¶

No /var/log/journal on SKY, RAIN, SUN or WIND. With Storage=auto that means none has ever persisted a journal; each holds roughly 14 volatile days, discarded on every reboot.

The check is the finding. systemctl --failed names RAIN alone, and SKY, SUN and WIND report systemd-journal-flush as active precisely because flushing is a successful no-op when the directory is absent. A check reporting green for the same condition it flags red elsewhere. The correct probe is test -d /var/log/journal, never the unit state.

RAIN's own failure — 2026-05-12 09:05:24 EDT, Main PID: 1046 (code=exited, status=0/SUCCESS) — is unrecoverable; the volatile journal reaches back only to 2026-08-26. Recorded as gone rather than guessed at. This is also why the original chronyd failure reason could not be retrieved.

Not fixed. All four have 25–28 GB free on a dedicated /var/log, so it is cheap, but it changes write behaviour on the DNS pair and RAIN's cause should be understood before it is potentially recreated on three more hosts.

3.3 New — unattended upgrades on both nameservers¶

dnf-automatic-install.timer is active on SKY and RAIN, applying package upgrades daily at about 06:0x, with no change window and no review. It is what put /usr/share/man/man1/wget.1.gz into the 2026-09-09 AIDE report.

Worth its own row. Note the near-miss it produced: denyall was hypothesised to be valid chrony 3.x syntax invalidated by an unattended upgrade, which would have made this a package-management finding. dnf history refused that — one chrony version has ever been installed. The hypothesis was wrong and the timer is still a real exposure.

3.4 New — WIND is a collector that stores nothing and is read by no one¶

Full enumeration 2026-09-09, read-only, no remedy applied:

Question	Answer
Who sends	SKY (2 connections), RAIN (3). SUN cannot — times out
What lands	DNS, DHCP and auth syslog. Nothing indexed since 2026-08-31
Why	Elasticsearch at 999/1000 shards; every write rejected HTTP 400
Who reads	Nobody. Kibana binds `127.0.0.1` only; 0 dashboards, 0 saved searches, 0 visualisations
Its blocked forward	SELinux blocks `syslogd_t → unreserved_port_t` on 5140 to CEDAR

The shard ceiling is self-inflicted: 518 primaries plus 481 replicas a single-node cluster can never assign equals exactly 999. The cluster is blocking its own ingestion with replicas it cannot use.

Two corrections to the earlier framing. WIND is not a possibly-orphaned destination — it is collecting today. And its blocked forward targets CEDAR, the live production store, not a destination with no owner. So semanage would close the GOV-039 (3) coverage gap rather than restore traffic nobody wants. Still not run — the decision belongs with the shard problem and the read question, and unblocking a forward into a store that already receives most of this data is not obviously right.

The remedy splits in two, and neither is today's work. Unblocking the forward is cheap and worth doing — the SELinux denial is protecting nothing. Fixing shard exhaustion is a separate problem. And if nothing reads WIND, the real question is whether it should exist at all rather than whether to repair it. That is an architecture decision and is routed to the architecture chat, not decided here.

DHCP is the one thing that is not duplicated. RAIN's /etc/rsyslog.d/30-dhcpd.conf sends local7.* to a local file and to WIND, then & stop — so CEDAR has never received DHCP logs at all (confirmed: 24 hours of CEDAR syslog contains named but zero dhcpd). No data is lost: /var/log/dhcpd.log is current on both nameservers with daily rotation. What is dead is the only searchable, centralised copy of DHCP lease history.

3.5 New — rule `100015` fires level 12 on the documented AIDE procedure¶

Ten level-12 LOLBin: /etc/shadow access attempt — T1003 OS Credential Dumping alerts on 2026-09-09, five more on 2026-09-02. All of them are AIDE. The audit record names it outright: comm="aide", exe="/usr/sbin/aide", AUID="dnsadmin", syscall 195 llistxattr — it listed extended attributes and did not read the file.

Only two days in fourteen carry this rule, and both are days an operator ran aide --update. The daily aide-check does not trigger it.

So following the documented post-change checklist generates critical credential-access alerts and drops the SOC score from 75 to 55. Our own post-change protocol trips our own detection.

The five on 2026-09-02 went unremarked at the time, and that is the sharper half: a rule already being ignored cannot detect what it was written for. That is a better statement of the alert-fatigue argument than VLN-010 managed, because it is not a projection about operator attention — it is a recorded instance of the alerts already being ignored, in this estate, by us.

The score movement was ours, not a regression. 75 → 55 was the detection working correctly on a benign trigger nobody had accounted for, and it was not the Webmin change. openvas: false was already false in a snapshot taken earlier the same morning and is pre-existing.

Same class as the 100010 curl exclusion.

3A. Fixed today, and the near-miss is the lesson — `GOV-045`¶

CEDAR held zero dhcpd documents over 24 hours beside 1,435 from named. The estate's central store had DNS and no DHCP at all. SKY was the worse of the two: its local7 reached no destination beyond the local file, not even WIND.

The tempting fix was wrong. rsyslog.conf includes rsyslog.d/*.conf at line 37 and its own defaults come after, at lines 47 and 66. DHCP uses local7, the facility RHEL reserves for boot logging. Deleting the & stop — described going in as a stray one-character defect — would have flooded /var/log/messages and /var/log/boot.log without bound. The & stop is load-bearing. The fix was to add the CEDAR forward above it, not remove it.

Verified against a zero baseline: 114 → 162 documents, +48 in 120 s, with rain at 78 and sky at 30 over 45 minutes.

No data was ever lost — /var/log/dhcpd.log is current on both hosts with daily rotation. What was missing was the central searchable copy, and RAIN's half of it was going to WIND, which has indexed nothing since 2026-08-31.

5.3 GCP firewall split — steps 1 and 2 DONE, step 3 HELD¶

Steps 1 and 2 are complete. Both broad rules were read in full first, and the narrow pair created from their real values:

Rule	Ports	Source	Target tag	Priority
`cedar-ingress-wdc-syslog`	`tcp:5140`	`192.168.120.0/23`	`cedar-logging`	800
`maple-ingress-wdc-agents`	`tcp:1514`, `tcp:1515`	`192.168.120.0/23`	`maple-monitoring`	800

Priority 800 sits ahead of the broad rules at 900, which is what makes step 3 a non-event rather than a cutover. Target tags match the existing objects exactly. Each description names its consumers, its senders and the dated GOV-045 addition — GOV-038's remedy applied to rules being created rather than only recorded about rules that exist.

They are not proven, and this is stated rather than glossed. The broad rules still permit the same traffic, so both feeds flow identically whether the narrow rules match or not. That is a signal true either way (M-13). The narrow rules are unproven until step 3 removes the range, and the ingestion figures recorded today are the baseline step 3 must be compared against, not evidence the new rules work.

Pre-step-3 baseline, measured 2026-09-09 17:09–17:13 UTC. This is the comparison target, not proof the new rules work:

Feed	Delta over 180 s	Per host, last 2 h
syslog `5140`	+93	`sky` 1926, `rain` 1583, `sun` 969
Wazuh `1514`	+4	`sky` 244, `rain` 242, `wind` 216, `sun` 116 (plus `maple` 192, `cedar` 104, `oak` 60)

All three syslog senders appear individually over a wide window, and all four WDC hosts are confirmed live Wazuh agents. After step 3 these figures must still move, per host, over a window of the same width — SKY and RAIN are bursty and a ten-minute window fails for the wrong reason.

SSH survives step 3 independently, verified from the objects rather than assumed: allow-onprem-via-vpn at priority 1000 grants tcp:22 to 192.168.120.0/23 and 192.168.124.0/24 in its own rule, so narrowing cedar-ingress and maple-ingress cannot cut off access.

Step 3 is held, and the reason is an open question¶

Grafana on 3000 and Prometheus on 9090 from the WDC LAN have no established consumer either way. maple-ingress carries both. If anyone reaches Grafana from the office, narrowing breaks it silently and with no obvious cause. Flow logging is disabled on both objects, so the evidence that would settle this was never collected. Answer it before step 3, not during.

Two other facts that change step 3¶

Narrowing cedar-ingress closes VLN-053 as a side effect. The object spans 22, 5140, 5601, 9200, 10000 and 9100, so it grants 9200 to the entire WDC LAN — the cloud layer permits every WDC host to reach CEDAR's unauthenticated Elasticsearch and only firewalld withholds it. That is VLN-053's two-layer disagreement confirmed from the firewall object itself. Step 3 is a security fix, not tidying.

Port 10000 is now dead config in both objects and should be REMOVED, not narrowed. Webmin was disabled on MAPLE and CEDAR today, so both rules permit a port to a service that no longer exists. That is exactly the accretion GOV-038 describes, with one difference: we created this instance ourselves, today. Removing it is cleanup rather than archaeology. A rule that outlives its service is how the next 10.8.0.0/28 begins.

Also: 9100 is scraped from MAPLE in 172.16.0.0/24 and does not need the WDC range at all.

Ownership is ours — gpus-vpc in gpus-infra, not a service project. SSH is unaffected via allow-onprem-via-vpn.

One dependency created today: DHCP now forwards to CEDAR on 5140 from both nameservers, so cedar-ingress-wdc-syslog carries the DHCP feed as well as general syslog. Its description says so.

4. Route to: FORMS¶

Okta Preview unverified.
The IT audit — Jack never replied; the HR batch is behind Taylor.
The 7-day signed-URL note to Tanu, Jack and Mike.

5. Two items with dates — do not lose these¶

*.us.gl3 expires 2026-09-17 — eight days from this document

Decision is DO NOT RENEW. Nothing presents this certificate. The action required is not a renewal, it is someone confirming on 2026-09-18 that nothing broke. If that confirmation is not scheduled, the decision is not actually made — it is just a renewal that did not happen.

The CA private key is world-readable on phoenix

/opt/puppet/dist/ca/, mode 0644. It signs every internal certificate in the estate. Found during the cert sweep and untouched since. This is the most serious permissions finding outstanding and it has no owner in any register.

6. State for the record¶

Retired: phoebe, pitta, bulbul. $613.59/month realised, $4.45 pending. Ledger v1.3.0. Retained snapshots are not saved and are not counted in the realised figure.

7. Method notes carried out of this session¶

These are the recurring shape, recorded because three separate instances landed in one week.

An empty result from the wrong path is not an absence. Four instances: an ss state filter that silently dropped IPv6-mapped sockets and made three live connections look like none; a grep "@" that could not match rsyslog's action(type="omfwd") form and so declared SKY's forward unexplained; a grep on /etc/aide.conf whose 2>/dev/null hid a permission denial and returned "no chrony rule"; and journalctl on a rotated journal reporting no entries for an event that certainly happened.
A check that passes for a reason unrelated to what it tests. systemd-journal-flush reads active on three hosts because nothing persists. Companion to M-13.
A partial explanation accepted as complete. The four-hour timezone offset explained the bulk of the WDC log lag, and the residual 17 and 21 minutes were attributed to low volume. That residual was GOV-041, visible in data already collected a day before it was found.
Verify a prediction rather than assuming it held. SKY's clock step was predicted to cross the hour and re-run two cron jobs. It crossed, and nothing re-ran. Checked in the journal, the anacron spool, the zone serial and the RRSIG rather than asserted.