VLN-009 — Passwordless root on all four WDC hosts via a shared backend SSH key¶

Classification: CONFIDENTIAL — Internal Use Only Document: security/finding-2026-08-05-wdc-passwordless-sudo.md · v1.1 · 2026-08-05 · GPUS-IT Status: OPEN — remediation designed, NOT YET APPLIED

1. Summary¶


Finding ID	VLN-009
Severity	Critical — highest open finding on the register
CVSS 3.1	9.9 — `AV:N/AC:L/PR:L/UI:N/S:C/C:H/I:H/A:H`
Affected	SKY, RAIN (primary + secondary DNS), SUN (Prometheus/Grafana), WIND (ELK)
Confirmed	2026-08-05, read-only against live hosts
Root cause	The 2026-06-23 sudo scope-down covered the GCP half only; the WDC half was never done
Remediation	Designed — see §6. Not applied. No sudoers file was modified during this assessment.

All four WDC on-prem hosts grant their SSH service account (ALL) NOPASSWD: ALL — unrestricted passwordless root. The same SSH key that reaches those accounts is mounted into three internet-facing Cloud Run services. Any code-execution or arbitrary-file-read defect in those services therefore yields immediate, unauthenticated-equivalent root on both organisational DNS servers.

On the CVSS score

9.9 is a chained score. The precondition is compromise of any one of the three key-holding Cloud Run services — gpus-status-backend, gpus-security-backend or gpus-soc-backend — all three internet-facing (allow-unauthenticated) and all three mounting gpus-status-backend-key. It is not a directly reachable unauthenticated path from the internet to DNS root.

Scored standalone, as a local privilege-escalation weakness (AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H), it would land at roughly 7.8 High. That scoring is not the one we carry, because it prices the privilege as though reaching the account were the hard part — and here it is not: the key is already mounted into three services that accept traffic from the internet. The chain is the realistic path, so the chained score is the one on the register.

The 9.9 reflects that once the precondition is met there is no further barrier at all — no password, no command restriction, no second factor — and the blast radius crosses a trust boundary (S:C) into infrastructure that is not part of the compromised component. Scored as a design weakness that removes every downstream control, not as a standalone remote-root vulnerability.

2. The exposure chain¶

Internet
   │  (allow-unauthenticated Cloud Run)
   ▼
gpus-status-backend / gpus-security-backend / gpus-soc-backend
   │  each mounts secret  gpus-status-backend-key:1  at /app/id_ed25519
   │  copied to /tmp/id_ed25519 (0600) at boot, used for an SSH fan-out
   ▼
SSH key  SHA256:nCsAfmkn0/qB3M9tQh9j1s4qSIhME3KCxE7k3x4cpM4  ("gpus-status-backend")
   │  present in authorized_keys on all 7 managed hosts
   ▼
dnsadmin@SKY   dnsadmin@RAIN   monitadmin@SUN   monitadmin@WIND
   │  (ALL) NOPASSWD: ALL
   ▼
root on both DNS servers, the metrics stack, and the log stack

The three cloud hosts (OAK, MAPLE, CEDAR) are not part of this chain: their cloudadmin account requires a password for sudo, remediated 2026-06-23 and re-confirmed 2026-08-05.

2.1 Correction to the reported scope¶

The exposure was originally described as running through the forms backend. That is not correct and the distinction matters for remediation:

Service	Internet-facing	Mounts SSH key
`gpus-status-backend`	yes	yes
`gpus-security-backend`	yes	yes
`gpus-soc-backend`	yes	yes
`gpus-forms-backend`	yes	no

forms-backend mounts no SSH secret and contains no SSH or paramiko code path. It is not part of this chain. The three services that are in the chain are the infrastructure dashboards.

3. Evidence — confirmed live, read-only¶

sudo -n -l on each host, 2026-08-05. Grant is in a dedicated drop-in file in every case:

Host	Account	File	Line
SKY	`dnsadmin`	`/etc/sudoers.d/dnsadmin:1`	`dnsadmin ALL=(ALL) NOPASSWD: ALL`
RAIN	`dnsadmin`	`/etc/sudoers.d/dnsadmin:1`	`dnsadmin ALL=(ALL) NOPASSWD: ALL`
SUN	`monitadmin`	`/etc/sudoers.d/monitadmin:1`	`monitadmin ALL=(ALL) NOPASSWD: ALL`
WIND	`monitadmin`	`/etc/sudoers.d/monitadmin:1`	`monitadmin ALL=(ALL) NOPASSWD: ALL`

Direct behavioural confirmation (sudo -n true, which succeeds only with a passwordless grant):

OAK    password required      MAPLE  password required     CEDAR  password required
SKY    PASSWORDLESS ROOT      SUN    PASSWORDLESS ROOT

Each host also carries correctly-scoped narrow grants (*-aide, gpus-nas-check, monitadmin-auditctl) — but sudoers applies last match wins, and the blanket NOPASSWD: ALL shadows them entirely. The narrow rules are already written; they are simply inert. This materially reduces remediation effort.

#	Observation	Risk
4.1	SUN carries `/etc/sudoers.d/dnsadmin` granting `dnsadmin` passwordless root, but no `dnsadmin` account exists on SUN. A latent grant that would activate silently the moment such an account is created.	Medium — dormant
4.2	The backend key appears twice in `authorized_keys` on SUN and WIND. Duplicate entries mean a revocation that removes one line leaves access intact.	Medium — revocation hazard
4.3	The operator laptop key is commented `dnsadmin@sky.wdc.us.gl3`, indicating it was generated on SKY and copied off-host, rather than generated on the workstation.	Low — provenance
4.4	CIS `Defaults` (`use_pty`, `log_input`, `log_output`, `logfile=/var/log/sudo.log`) are set on SKY/RAIN but absent on SUN/WIND — sudo activity on the monitoring hosts is not session-logged.	Medium — reduced forensics
4.5	`/etc/sudoers.d/gpus-nas-check` on WIND is 0 bytes.	Low — drift
4.6	One key pair is shared across all three backends and all seven hosts, so per-service revocation is impossible.	High — no isolation

5. What the backends actually need¶

Every privileged command the three services issue over SSH, extracted from status-backend/app.py, security-backend/app.py and soc-backend/app.py:

auditctl -s                                     firewall-cmd --get-active-zones
ausearch -ts today                              firewall-cmd --list-all
fail2ban-client status sshd                     lastb
grep -c 'Ban ' /var/log/fail2ban.log            ss -tlnp
grep -c 'changed|added|removed' aide.log        stat -c %Y /var/log/aide/aide.log
grep 'Failed password' /var/log/secure          ls /mnt/nas-backup/<host>/
grep 'CIS' /var/log/asset-inventory.log

Every one is read-only telemetry. Two consequences:

No DNS operation is performed over SSH. Nothing touches named, rndc, zone files or dnssec-signzone. Root on a DNS server is not required by any dashboard.
No backup write is performed over SSH. Backups are written by a root cron job on each host (/usr/local/bin/gpus-portal-backup.sh, 02:30 daily, installed into the root crontab by gpus-backup-cron-install.sh). The backend only lists /mnt/nas-backup/<host>/ to read the latest timestamp.

The entire legitimate privilege requirement is therefore twelve read-only commands. Nothing needs write access, and nothing needs ALL.

6. Planned remediation — NOT YET APPLIED¶

Design only. No change has been made. Full plan, sequencing and lockout controls are in §7.

Target state, mirroring the 2026-06-23 GCP remediation exactly:

<account> ALL=(ALL) ALL                          # password required, interactive admin
<account> ALL=(root) NOPASSWD: <narrow aliases>  # only the telemetry commands

7. Remediation plan¶

7.1 Preferred design — ACL for reads, narrow sudo for the rest¶

Seven of the twelve commands only need to read a root-owned file. Those do not need sudo at all: grant the service account read access via a group or POSIX ACL on the specific log paths, and drop them from sudoers entirely. That leaves five genuinely privileged binaries:

Needs sudo (privileged binary)	Solved by read access instead
`auditctl -s`, `ausearch -ts today`	`/var/log/aide/aide.log`
`fail2ban-client status sshd`	`/var/log/fail2ban.log`
`firewall-cmd --get-active-zones`, `--list-all`	`/var/log/secure`
`lastb`, `ss -tlnp`	`/var/log/asset-inventory.log`, `/mnt/nas-backup/<host>/`

Smaller sudo surface, no wildcards, and file reads become auditable through normal filesystem permissions.

7.2 Fallback — like-for-like sudoers scope-down¶

If ACL changes are deferred, use Cmnd_Alias drop-ins only. Prefer exact commands; avoid * in argument position — the existing dnsadmin-aide rule (/usr/bin/grep * /var/log/aide/aide.log) is looser than it looks and should be tightened while in flight.

7.3 Sequencing — additive first, and why it matters¶

The real risk is a monitoring blackout, not just an admin lockout

Removing NOPASSWD: ALL immediately breaks every backend SSH command that is not explicitly re-granted. The dashboards would silently degrade — 2>/dev/null || echo 0 in the backend code means a denied sudo returns a plausible zero rather than an error. A partially-scoped ruleset therefore produces green dashboards reporting nothing, which is worse than an outright failure. Never remove the blanket grant in the same step as adding the narrow ones.

Inventory-freeze. Confirm the §5 command list against the deployed revisions, not the repo — a drifted revision may issue commands the repo no longer contains.
Add the narrow grants and/or ACLs alongside the existing blanket grant. No loss of access; blanket grant still shadows everything.
Verify each backend still returns real data — compare every dashboard tile against its pre-change value. Confirm no tile silently reads zero.
Remove NOPASSWD: ALL on one host only.
Re-verify, then roll to the remaining hosts one at a time.
Re-confirm with sudo -n true (must fail) and sudo -n -l (must list only the narrow set).

7.4 Host order¶

RAIN → SUN → WIND → SKY. RAIN is the secondary DNS server: if sudo breaks there, SKY still answers queries. SKY, the primary and the only host that can re-sign the zone, goes last. Never SKY first.

7.5 Lockout controls — mandatory¶

Keep a root shell open on a second channel for the entire change window, and do not close it until sudo -n -l has been verified. This is the single control that turns a lockout into an inconvenience.
Validate before install: author to a temp file, visudo -cf <file>, then install. A syntax error anywhere in /etc/sudoers.d/ disables sudo for all users on that host.
Confirm out-of-band access first (console/IPMI/hypervisor) and verify it works before touching the first host.
Know the root password, or confirm single-user boot is available, before starting.
One host at a time, with verification between. Never parallelise across SKY and RAIN — that risks both DNS servers simultaneously.
Change window outside DNS-critical hours, with the DNS runbook to hand.

7.6 Follow-on items¶

Rotate gpus-status-backend-key after the scope-down, and issue per-service keys so a single compromised service no longer implies all seven hosts (§4.6).
Remove the duplicate authorized_keys entries on SUN and WIND (§4.2).
Remove the latent /etc/sudoers.d/dnsadmin from SUN (§4.1).
Apply the CIS sudo Defaults to SUN and WIND (§4.4).
Remove or populate WIND's empty gpus-nas-check (§4.5).
Consider restricting the backend key with command= / restrict in authorized_keys, capping what the key can invoke regardless of sudoers.

8. References¶

security/vuln/tracker.md — VLN-009 register entry
2026-06-23 GCP sudo remediation — the pattern this mirrors
servers.py — the seven-host SSH fan-out
status-backend/cloudbuild.yaml:36, security-backend/cloudbuild.yaml:26, soc-backend/cloudbuild.yaml:36 — secret mount