Change Procedure¶

Version 1.1 | 2026-09-09 | Classification: CONFIDENTIAL — Internal Use Only

Normal Change — Step by Step¶

Step 1 — Document the Change Request¶

Before touching any system, record the following:

CHANGE REQUEST
──────────────────────────────────────────────────────
CR ID:            CR-YYYY-NNN
Date Submitted:   YYYY-MM-DD
Submitted By:     [Name]
Target System(s): [SKY / RAIN / SUN / WIND / GCP]
Planned Date/Time: YYYY-MM-DD HH:MM UTC
Change Type:      Normal / Emergency

Description:
[What is being changed and why]

Risk Assessment:
  Impact if successful:   [Low / Medium / High]
  Impact if it fails:     [Low / Medium / High]
  Probability of failure: [Low / Medium / High]
  Overall risk:           [Low / Medium / High]

Test Plan:
[How will you verify the change worked correctly]

Rollback Plan:
[Exact steps to undo the change if it fails]

Approval:
  IT Manager:  ________________________  Date: ____________
──────────────────────────────────────────────────────

Step 2 — Obtain Approval¶

Send the change request to the IT Manager at least 24 hours before the planned implementation window. Do not proceed without written or email approval.

Step 3 — Pre-Change Snapshot (on-prem)¶

Take an ESXi snapshot of the affected VM(s) immediately before applying the change:

ESXi Console → Right-click VM → Snapshots → Take Snapshot
Name: pre-CR-YYYY-NNN-YYYY-MM-DD
Description: Pre-change snapshot for CR-YYYY-NNN

Step 4 — Apply the Change¶

Implement the change exactly as documented in the change request. If anything unexpected occurs, stop and initiate the rollback plan immediately — do not improvise.

Step 5 — Run the Post-Change Checklist¶

# 1. Update AIDE baseline. NEVER chain the promotion with &&.
#    aide --update exits NON-ZERO (7) whenever it finds drift, so
#    "aide --update && mv" promotes ONLY when there was nothing to promote.
#    Run it, confirm the new database exists, then promote unconditionally.
sudo aide --update
sudo ls -l /var/lib/aide/aide.db.new.gz
sudo cp -p /var/lib/aide/aide.db.gz /var/lib/aide/aide.db.gz.bak.CR-YYYY-NNN
sudo mv -f /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz

# 2. Log the change. Match the pipe-delimited format already in the file.
#    <ISO8601 UTC> | <host> | <role> | operator=<user> | ticket=<id> | <what and why>
echo "2026-01-01T00:00:00Z | HOST | ROLE | operator=NAME | ticket=CR-YYYY-NNN | DESCRIPTION" \
    | sudo tee -a /var/log/asset-inventory.log

# 3. Re-sign DNSSEC (SKY only). Do NOT run dnssec-signzone by hand.
sudo /etc/cron.weekly/dnssec-resign

Step 5 corrected 2026-09-09 — the previous version taught three known defects

Anyone following the earlier text produced a silently-unchanged baseline and a re-signed zone that reverted a security remediation. All three were already recorded as findings elsewhere before this page was fixed.

1. aide --update && sudo mv never promotes when it matters. aide --update exits non-zero on any detected change (1 new, 2 removed, 4 changed; 7 is a normal drift run), so && runs the mv only when nothing changed. Every run that actually found drift left an orphaned aide.db.new.gz and an untouched baseline. Observed on both nameservers: SKY sat on an April baseline and RAIN on a March one for months while the checklist was being followed correctly.

2. The old dnssec-signzone line generated a fresh random NSEC3 salt every run. VLN-014 pinned a specific salt on 2026-08-17; a hand-run of that command silently overwrites it and rewrites the whole NSEC3 chain, reverting a documented remediation.

3. -N INCREMENT reads the serial from the input file and writes input+1 to the output, never modifying the input. Twenty-seven weekly runs from 2026-03-04 therefore emitted the identical served serial, RAIN saw no change and never transferred, and the secondary served stale signatures. /etc/cron.weekly/dnssec-resign was rewritten on 2026-09-02 to bump the source serial, use -N KEEP, gate the reload on named-checkzone, check signing and reload exit codes separately, and verify the transfer to RAIN. Call that script; do not reimplement it.

Step 6 — Verify¶

Run the test plan documented in the change request. Confirm expected outcomes. If verification fails, execute the rollback plan and notify the IT Manager.

Step 7 — Update the Change Log¶

Add an entry to the Change Log and close the change request with outcome (Success / Rolled Back / Partial).

Emergency Change — Step by Step¶

Call the IT Manager immediately. State the incident, the proposed change, and the risk.
Implement the change with verbal approval (or immediately if unreachable and service is down).
Run the Post-Change Checklist as soon as the emergency is resolved.
Complete full change documentation within 2 hours.
Add a change log entry tagged [EMERGENCY].
Review at next team meeting.

Standard Change — Step by Step¶

Confirm the task matches a Standard Change Template exactly.
Follow the template procedure without deviation.
Run the Post-Change Checklist.
Log to /var/log/asset-inventory.log with the template name referenced.

Procedure · v1.1 · 2026-03-14 · GPUS-IT · Classification: CONFIDENTIAL — Internal Use Only