Change Procedure¶

Version 1.0 | Classification: CONFIDENTIAL — Internal Use Only

Normal Change — Step by Step¶

Step 1 — Document the Change Request¶

Before touching any system, record the following:

CHANGE REQUEST
──────────────────────────────────────────────────────
CR ID:            CR-YYYY-NNN
Date Submitted:   YYYY-MM-DD
Submitted By:     [Name]
Target System(s): [SKY / RAIN / SUN / WIND / GCP]
Planned Date/Time: YYYY-MM-DD HH:MM UTC
Change Type:      Normal / Emergency

Description:
[What is being changed and why]

Risk Assessment:
  Impact if successful:   [Low / Medium / High]
  Impact if it fails:     [Low / Medium / High]
  Probability of failure: [Low / Medium / High]
  Overall risk:           [Low / Medium / High]

Test Plan:
[How will you verify the change worked correctly]

Rollback Plan:
[Exact steps to undo the change if it fails]

Approval:
  IT Manager:  ________________________  Date: ____________
──────────────────────────────────────────────────────

Step 2 — Obtain Approval¶

Send the change request to the IT Manager at least 24 hours before the planned implementation window. Do not proceed without written or email approval.

Step 3 — Pre-Change Snapshot (on-prem)¶

Take an ESXi snapshot of the affected VM(s) immediately before applying the change:

ESXi Console → Right-click VM → Snapshots → Take Snapshot
Name: pre-CR-YYYY-NNN-YYYY-MM-DD
Description: Pre-change snapshot for CR-YYYY-NNN

Step 4 — Apply the Change¶

Implement the change exactly as documented in the change request. If anything unexpected occurs, stop and initiate the rollback plan immediately — do not improvise.

Step 5 — Run the Post-Change Checklist¶

# 1. Update AIDE baseline
sudo aide --update && sudo mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz

# 2. Log the change
echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [$(hostname)] CR-YYYY-NNN: <description> — $(whoami)" \
    | sudo tee -a /var/log/asset-inventory.log

# 3. Re-sign DNSSEC (SKY/RAIN zone changes only)
dnssec-signzone -A -3 $(head -c 6 /dev/random | od -An -tx1 | tr -d ' \n') \
    -N INCREMENT -o wdc.us.gl3 -t /var/named/wdc.us.gl3.db
rndc reload

Step 6 — Verify¶

Run the test plan documented in the change request. Confirm expected outcomes. If verification fails, execute the rollback plan and notify the IT Manager.

Step 7 — Update the Change Log¶

Add an entry to the Change Log and close the change request with outcome (Success / Rolled Back / Partial).

Emergency Change — Step by Step¶

Call the IT Manager immediately. State the incident, the proposed change, and the risk.
Implement the change with verbal approval (or immediately if unreachable and service is down).
Run the Post-Change Checklist as soon as the emergency is resolved.
Complete full change documentation within 2 hours.
Add a change log entry tagged [EMERGENCY].
Review at next team meeting.

Standard Change — Step by Step¶

Confirm the task matches a Standard Change Template exactly.
Follow the template procedure without deviation.
Run the Post-Change Checklist.
Log to /var/log/asset-inventory.log with the template name referenced.

Procedure · v1.1 · 2026-03-14 · GPUS-IT · Classification: CONFIDENTIAL — Internal Use Only