Backlog — enumeration sessions 2026-08-13 to 2026-08-17¶
Classification: CONFIDENTIAL — Internal Use Only Document:
priorities/backlog-2026-08-17.md· v1.0 · 2026-08-17 · GPUS-IT
Source text for the register load of 2026-08-17. Items are provisional
identifiers only; real IDs are assigned on load and recorded in the mapping
document. Nothing here is done. Where no acceptance test is stated, the row
is INCOMPLETE and must not be filled with a plausible guess.
Routing:
| Provisional | Register | Document |
|---|---|---|
| B-01 – B-14 | Security findings | security/vuln/tracker.md |
| B-15 – B-24 | Governance / tracking gaps | governance/gap-register.md |
| B-25 – B-38 | Program work items | priorities/gpus-it-priorities.md |
| M-01 – M-10 | Method learnings | governance/gap-register.md, new section, unnumbered and status-less |
Security findings¶
B-01 — CRITICAL. vmstorage (RS1619xs+, 192.168.120.51) running a degraded
RAID 6: designedDiskCount 6, normalDevCount 5, disk_failure_number 1,
spares [], repair_action degrade, data scrubbing unavailable
(reason: abnormal). Three disks sit in not_use (sdeb, sdec, sdfa), all
reporting SMART normal. This is the NFS datastore for every WDC VM, mounted by
fire, water and flower. Nothing in monitoring surfaced it and no artifact
mentions it. Evidence: Synology DSM API, 2026-08-17. Acceptance: array returns
normal with a spare assigned and scrubbing available.
B-02 — CRITICAL. Plaintext vendor credentials in
modules/relay/files/etc/pushtab lines 1–6. Six rows, format
name user password endpoint — field 3 is a plaintext password on every row,
covering payment, payroll and CRM vendors (litle, convio, adp, target). Last
commit 2019-01-24. Deployed to /opt/gp/etc/pushtab on every storehouse node.
Repo has 879 commits and five known working copies. Two values also transited
terminal output on 2026-08-17. Not closed by the Rocky migration — deleting
the file does not invalidate the credentials. Acceptance: all six rotated at the
vendor or the accounts disabled vendor-side, confirmed per vendor.
B-03 — HIGH. puppetCrypt exists (125 GPG-encrypted secrets) and pushtab
bypasses it. The capability was present; the file ignored it. Acceptance: no
plaintext credential in the control repo.
B-04 — HIGH. vmstorage has no backup or replication task.
SYNO.Backup.Task returns 0, SYNO.Backup.Repository repo_list is empty,
CloudSync API not registered. Recorded as configured-absent, not unchecked.
Compounds B-01. Evidence: Synology DSM API, 2026-08-17.
B-05 — HIGH. IPsec pre-shared keys for both VPN peers returned in plaintext
by the Meraki thirdPartyVPNPeers endpoint and transited terminal output on
2026-08-13. Durable handling rule: any read of Meraki VPN config exposes PSKs.
Acceptance: both rotated, coordinated with the GCP side, tunnel verified up.
B-06 — HIGH. SMBv1 enabled on all three Synology units
(smb_min_protocol = 1, the lowest selectable value, corresponding to SMB1/NT1)
with server signing off (0) on all three. NTLMv1 is off. duck and grebe are
Windows SMB proxies. Evidence: Synology DSM API v3, 2026-08-17. Acceptance:
minimum protocol raised, signing enabled, verified per unit.
B-07 — HIGH. Key material in the control repo, identified by filename only —
contents never read: ca/ 81 private-key files, dnssec/ 1 .private,
htpasswd/ 18 hash files. The repo lives in a project scheduled for deletion.
Deleting the repo copy does not revoke anything still trusted. Acceptance: each
item determined live or historical; anything live rotated or relocated.
B-08 — MEDIUM. Undocumented second VPN peer: Target, 206.31.252.84,
IKEv1 (deprecated), subnet 10.5.142.0/24. Appears in no artifact, no brief, no
diagram. Evidence: Meraki API, 2026-08-13. Acceptance: owner identified, tunnel
justified in writing or removed.
B-09 — MEDIUM. Zero 2FA across all three Synology units — 2fa_status: false
on every user on every unit. The admin account is present and not renamed,
though expired/disabled. Evidence: Synology DSM API, 2026-08-17.
B-10 — MEDIUM. Meraki org 395909: saml.enabled false, zero IdPs
configured, all four admins authenticationMethod: Email. The W2 SSO migration
has not started rather than being partially done. Only one admin holds an API
key. Evidence: Meraki API, 2026-08-13.
B-11 — MEDIUM. synstorage volume_1 at 93.1% and volume_3 at 82.3%,
both status attention — 13.44 TB and 187.64 TB respectively. Media/Video data
with no documented backup. Evidence: Synology DSM API, 2026-08-17.
B-12 — MEDIUM. rooster.cloud.us.gl3 cannot compile in production:
archive::backup has its entire body commented out (lines 5–17) in the
production environment but is defined normally in seed and testing. rooster is
absent from PuppetDB, which is consistent. Children archive::backup::files and
archive::backup::cron still exist and are now unreachable. Third instance of
LOADED ≠ REACHABLE in this program. Evidence: Stage 3 manifest read, 2026-08-17.
B-13 — MEDIUM. dataflow.us.gl3 resolves to merlin
(zones/us.gl3.zone:61), which has no VM in any project and is ASSERTED-STALE.
Every live vendor script posted status there. Downgraded: the transfer stack is
DECOMMISSION per R. Chhetry 2026-08-17, so this is evidence supporting the
decommission rather than an active incident.
B-14 — LOW. wdc-wap-5 (MR57, WDC) status alerting. Evidence: Meraki API,
2026-08-13.
Governance / tracking gaps¶
B-15. inventory.yaml holds 82 entries against 481 deduped live entities.
gpusa has zero entries against 174 live entities. gpus-it has 9 against 57.
last_updated: 2026-06-26. The original GOV-001 framing — competing
single-source-of-truth claims at hostregistry/index.md:13 against three
documents asserting inventory.yaml — is accurate but secondary. The primary
finding is that the declared source of truth covers roughly a fifth of the
estate, and that is why the coverage gate has been passing: it validates only
what is declared, and nothing declares the largest location. Evidence: Stage 2
A1 census, 2026-08-17. Acceptance: inventory.yaml contains an entry for every
entity in enumerated.yaml, or a dated exception with an expiry for each
omission. This row rewrites GOV-001; retain the current GOV-001 wording as a
superseded note.
B-16. PuppetDB enforcement gap. 11 certnames against 39 FQDN node definitions. Thirteen RUNNING VMs with node definitions have never produced a retained catalog, including phoenix itself. Confidence and enforcement are different axes: a DERIVED function statement on a host with no retained catalog is a hypothesis, not evidence. This must gate disposition, not merely annotate it. Evidence: Stage 3, 2026-08-17.
B-17. Five nodes declare puppetMaster => raven.cloud.us.gl3. raven is a
DNS A record at 10.1.96.25 with no backing host. Those nodes have been
applying against a dead master — managed-in-name-against-nothing, a third state
that is neither managed nor unmanaged. Classified ASSERTED-STALE. Evidence:
Stage 3, 2026-08-17.
B-18. The Puppet control repo does not reproduce the running configuration.
Untracked but load-bearing: production/environment.conf (defines modulepath),
hiera.yaml, an emacs autosave #environment.conf#, and five vendored modules
(apt, concat, inifile, stdlib, systemd). Mirror /etc/puppetlabs/code as a
filesystem capture, not a git clone. Evidence: Stage 3 Task 1a, 2026-08-17.
B-19. Three repos on phoenix across two CSR projects, all on deprecated
Cloud Source Repositories: gpus-puppet and gpus-dist in
gpus-it-infrastructure (scheduled for deletion), and /opt/puppet/static
served as puppet:///static/ from gpusa-it-infrastructure/gpusa-puppet,
branch testing, last commit 2019-08-30, 39 GB — unassessed. Mirror before any
teardown work touches those projects. Evidence: Stage 3b, 2026-08-17.
B-20. Twelve power devices (PDUs/UPS) were never enumerated by any Stage 1
surface; six have 192.168.122.x management IPs that nothing probed. Firmware
and credentials unknown. This is an enumeration gap, not an estate fact — do not
infer state from absence. Evidence: Stage 2 A3, 2026-08-17.
B-21. Join-key constraint, learned in both directions. Hostname is not a
reliable join key in this estate. False absence: 21 Meraki devices were reported
DECLARED_NOT_FOUND because inventory uses descriptive names (wdc_stack_0,
mdec_wap_garage) while Meraki reports operational ones (WDC-STACK-0-NOPOE,
mdec garage); re-matching on serial and MAC, both present in
inventory.yaml, resolved all 21 and would otherwise have inflated
DECLARED_NOT_FOUND from 23 to 44. False presence: forms.us.gl3 matched
gpus_forms_frontend, whose FQDN is forms.greenpeace.us — conflating the
legacy portal with its replacement and hiding a retirement candidate behind it.
Any reconciliation control must join on serial/MAC for devices and IP for DNS.
Attach to GOV-004 as a design constraint. Evidence: Stage 2, 2026-08-17.
B-22. Property disagreements between inventory.yaml and live state.
Presence was never the issue; the declared attributes are wrong.
| Entity | Field | Declared | Live |
|---|---|---|---|
| fire | vms | [] |
sky, rain, sun, wind |
| water | hypervisor | ESXi 6.7 | ESXi 8.0.3 (build 24022510) |
| water | vms | ["ocean"] |
none |
| flower | hypervisor | ESXi 6.8 (no such release) | ESXi 6.7.0 (build 8169922) |
| flower | vms | [] |
ocean, desert, river, star |
| fire / water / flower | hardware_model | "" |
R610 / R660xs / R630 |
| vmstorage | model | "Synology" | RS1619xs+ |
| solstorage | model | "Synology" | DS1823xs+ |
| synstorage | — | not declared at all | SA3600, 3 volumes |
| SKY / RAIN / SUN / WIND | project | gpus-infra | on-prem WDC VMs on fire |
fire declared as hosting nothing while carrying all four core WDC servers is the
sharpest: any blast-radius reasoning from that field is wrong, and that field is
what a coverage gate reads. Compute instances showed no disagreement on ip,
machine_type or state. Evidence: Stage 2 Part B, 2026-08-17. Justifies
widening VLN-013's acceptance test to cover inventory.yaml:146-152 and
downstream generated output, not only VLN-004.
B-23. Orphan count superseded twice. 35 (Stage 1b raw A-record cross-reference) → 52 (Stage 2 matcher, which did not know the estate spans two GCP projects for Puppet purposes) → 50 (Stage 3, short-name method accounting for the sibling project). Record 50 with the method stated; both prior figures superseded. Evidence: Stage 3, 2026-08-17.
B-24. Snapshot configuration unknown on all three Synology units — error 403, the querying account lacks rights to that API. Unknown, not absent. Blocks assessing the severity of B-04. Evidence: Synology DSM API, 2026-08-17.
Program work items¶
B-25. Consolidate fire and flower onto water. water: Xeon Gold 5416S,
PowerEdge R660xs, 63.5 GB RAM at 3.8%, 1.66 TB datastore at 0.1%, zero VMs.
fire: Xeon E5640 (2010 silicon), R610, 48 GB at 70%, hosts all four core
servers. flower: Xeon E5-2697 v3, R630, 128 GB at 11.6%, local datastore 80.6%
full. Combined RAM in use across fire and flower is ~48 GB against water's
63.5 GB. All three already mount the same vmstorage NFS datastore, so compute
moves without storage moving. Retires two EOL 6.7 hosts onto supported hardware
already owned. RAM headroom is thin — per-VM allocations needed before
committing. Evidence: authenticated vSphere API, 2026-08-13.
B-26. WS2025 golden image family is a dependency, not a future item.
inventory.yaml declares duck, grebe and nfs-gw as REPLACE targeting Windows
Server 2025, while the golden-image program currently plans a single Rocky 9
family. A second image lane is required. Evidence: Stage 2 Part B, 2026-08-17.
B-27. magpie contradiction. inventory.yaml declares disposition: keep,
role_status: confirmed-data-team, os: rhel-8, os_eol: 2029-05-31, live
RUNNING — in gpus-it-infrastructure, which is committed to empty and retire
entirely. Either the commitment has an unwritten exception or magpie needs a
destination. Decision for R. Chhetry and Rob MacMillan. Evidence: Stage 2 Part
B, 2026-08-17.
B-28. Seven unmanaged gpus-it hosts unexplained: duck, falcon, grebe, gull,
kestrel, nfs-gw, woodpecker. Running, never Puppet-managed, and nothing in
either repo accounts for them. falcon, gull and kestrel appear only as DNS
records; grebe, nfs-gw and woodpecker appear nowhere. falcon's public record
(35.223.86.92, 2026-07-27) is the newest content in gpus-dist. Evidence:
Stages 3 and 3b, 2026-08-17.
B-29. Two undocumented Meraki networks: DC Apartments (4 APs — MR33×3, MR52×1) and Major Gifts (systems-manager only), plus three unassigned APs (MR32×2, MR33×1). The working brief says three sites; there are five networks and 32 devices, 29 assigned. Evidence: Meraki API, 2026-08-13.
B-30. Meraki WDC edge is an MX95 HA pair with warm spare enabled
(primarySerial Q2XN-V4XE-UQKX), firmware wired-26-1-5. The asset registry
says MX100; the registry is wrong. WAN1 virtual IP 38.140.146.68 matches the
documented VPN endpoint; the physical MXs hold .66 and .67. Evidence: Meraki
API, 2026-08-13.
B-31. Vendor transfer stack — DECOMMISSION. Evidence: R. Chhetry decision,
2026-08-17: stale code, not required, removed with the move to Rocky Linux.
Covers the relay module, the transfer crons and the storehouse hosts. This is
a decision, not a verification — do not record as done. Premise corrections
recorded with it: there are 27 cron resources but only 13 potentially-active
transfers, because 13 are declared twice (ensure => present under
if($live == true), ensure => absent in the else branch) and pushPSI is
unconditionally absent. Three scripts do not do what their names say —
pushFacter performs no transfer at all and only downcases filenames locally;
pushFPR exits 0 at line 7 ("Turning this script off for now") and has fired
every 30 minutes doing nothing; pushPSI is explicitly deactivated. Scripts
live in the Puppet control repo at modules/relay/files/sbin/ with -rooster
and -whistler variants; sourceselect => first picks the hostname variant and
whistler is the RUNNING one. Endpoint routing comes from pushtab.
B-32. Stage 3 result: 24 of 25 gpusa VMs moved UNKNOWN → DERIVED. Only quail
remains UNKNOWN (TERMINATED, no node definition). Across all 44 node
definitions: 38 DERIVED, 5 ASSERTED-STALE, 1 ASSERTED. kingfisher resolved to
DERIVED via the testing modulepath, where postgresql is defined —
puppetEnvironment => testing. Subject to B-16: DERIVED is not enforced.
Evidence: Stage 3, 2026-08-17.
B-33. The estate spans two GCP projects for Puppet purposes. magpie and
kingfisher have node definitions and live in gpus-it-infrastructure, not
gpusa. Comparing node definitions against gpusa VMs alone over-reports orphans
by two. Evidence: Stage 3, 2026-08-17.
B-34. gpusa UNKNOWN count is 154, not 146, as read 2026-08-17.
B-35. gpus-dist staleness profile: 900 of 946 files (95%) are more than
three years old; 800 were last touched in 2017. The sole current content is DNS
zones — zones/greenpeaceusa.org*.zone at 2026-07-27 is the newest file in the
repo. DNS is the consistent exception to staleness across both repos. Evidence:
Stage 3b per-file provenance, 2026-08-17.
B-36. duck IP contradiction confirmed from a second independent source.
gpus-dist forward and reverse zones both say 10.1.96.40; the live VM is
10.1.96.46; inventory.yaml says .46. Cross-reference to VLN-019 as
second-source corroboration. Evidence: Stage 3b, 2026-08-17.
B-37. Enumeration surfaces to add to any future reconciliation control:
Pub/Sub topics, Cloud Scheduler jobs, host-resident services, and power devices
were in no Stage 1 surface. Four declared Pub/Sub and Cloud Scheduler entities
plus one Pub/Sub subscription (gpus_forms_clamav_worker_sub) could not be
checked in either direction. Evidence: Stage 2, 2026-08-17.
B-38. Seven entities declared with no identifying data and never observed
live: gl5_firewall (no serial, no MAC, no IP), synology_controller,
visuals_storage_exp_1 through _3, vmware_storage. Unresolved in both
directions. Evidence: Stage 2 A3, 2026-08-17.
Method learnings¶
Constraints on future controls, not findings. Unnumbered, no status, no owner.
M-01 — LOADED ≠ REACHABLE. Three instances in this program: VLN-010's Wazuh rules, rooster's commented-out class, and the transfer crons firing on schedule doing nothing. This is the estate's characteristic failure — a control present, scheduled and running while doing nothing.
M-02 — Permission-denied is not unreachability. An SSH timeout is a gap to route around; a 403 means the enumerator is blind and must halt. Recording a denial as UNREACHABLE produces a file that looks complete because it ran to the end.
M-03 — Exit codes carry no information for gcloud list verbs. All six access probes returned exit 0, denials included, with the error only on stderr. stderr capture is the sole discriminator.
M-04 — --quiet is required on read-only gcloud calls, and forbidden on
mutating ones. GCP returns PERMISSION_DENIED for a disabled API as well as an
IAM denial, and gcloud converts that into an enable-and-retry prompt — a project
mutation offered inside a read stage. On mutating verbs --quiet auto-confirms.
M-05 — API_DISABLED is not an IAM denial. An IAM denial hides resources that may exist; a disabled API means that resource type has no enumeration path on that project.
M-06 — Recorded absence, never silent absence. A blocked project gets an explicit BLOCKED row with the probe result and date.
M-07 — Confidence and enforcement are different axes. Reading a manifest tells you what is declared, not whether it ever applied.
M-08 — Counts supplied from memory rather than a live read are provisional. Four were overturned during these sessions: the orphan count twice, the gpusa UNKNOWN count, and the unlisted Cloud Run service count. Every count in a register needs a source.
M-09 — Enumerate fully before classifying. The moment classification starts, the list looks finished and enumeration quietly stops.
M-10 — UNKNOWN must be the default and must survive to the end. The honest figure at Stage 2 was 50.0%. Every prior artifact implied near-complete knowledge.
Change log¶
| Version | Date | Author | Change |
|---|---|---|---|
| v1.0 | 2026-08-17 | R. Chhetry / Claude | Initial backlog. 38 items (B-01–B-38) and 10 method learnings (M-01–M-10) from enumeration Stages 1, 1b, 1c, 2, 3 and 3b. |