Skip to content

Blue Team Drills

Classification: CONFIDENTIAL — Internal Use Only Document: security/redblue/blue-team-drills.md · v1.0 · 2026-03-16 · GPUS-IT

Drill 1 — Brute force SSH detection

Objective: Verify Fail2ban detects and bans attacking IPs within threshold (5 failures / 10 min).

Pre-requisites

  • Fail2ban active on target server
  • Test IP not already in whitelist
  • Coordinate with Director of Cyber Security before running

Verification steps

# 1. Check Fail2ban is active and monitoring sshd
sudo fail2ban-client status sshd

# 2. Verify jail configuration
sudo fail2ban-client get sshd maxretry
sudo fail2ban-client get sshd findtime
sudo fail2ban-client get sshd bantime

# 3. Review recent bans
sudo grep "Ban " /var/log/fail2ban.log | tail -20

# 4. Check failed auth attempts in secure log
sudo grep "Failed password" /var/log/secure | tail -20

# 5. Verify ban is applied in firewall
sudo firewall-cmd --list-rich-rules | grep -i reject

# 6. Check ELK is ingesting Fail2ban events
curl -s http://192.168.120.4:9200/fail2ban*/_count 2>/dev/null | python3 -m json.tool

Pass criteria

  • IP banned within 60 seconds of 5th failed attempt
  • Ban persists for configured duration
  • Fail2ban log shows Ban <IP> entry
  • firewall-cmd shows rich rule rejecting banned IP

Drill 2 — AIDE file integrity detection

Objective: Verify AIDE detects unauthorized changes to monitored files within the 24-hour cron cycle.

Steps

# 1. Record current AIDE baseline hash
sudo sha256sum /var/lib/aide/aide.db.gz

# 2. Make a test change to a monitored file (use a non-critical path)
sudo touch /etc/aide-test-$(date +%Y%m%d) && echo "test" >> /etc/aide-test-$(date +%Y%m%d)

# 3. Run AIDE check manually
sudo aide --check 2>&1 | tee /tmp/aide-drill-$(date +%Y%m%d).txt

# 4. Verify change detected
grep -E "changed|added|removed" /tmp/aide-drill-$(date +%Y%m%d).txt

# 5. Verify auditd captured the write
sudo ausearch -f /etc/aide-test-$(date +%Y%m%d) -ts today

# 6. Clean up test file
sudo rm /etc/aide-test-$(date +%Y%m%d)

# 7. Update AIDE baseline
sudo aide --update && sudo mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz

# 8. Log the drill
echo "$(date) [DRILL] AIDE file integrity drill completed — PASS" >> /var/log/asset-inventory.log

Pass criteria

  • AIDE detects the test file addition
  • auditd records the write syscall
  • No unauthorized changes present in baseline after cleanup
  • AIDE baseline updated successfully

Drill 3 — DNS failover (SKY → RAIN)

Objective: Verify RAIN assumes primary DNS/DHCP within acceptable RTO when SKY is unavailable.

Steps

# 1. Baseline — verify both servers resolving
dig @192.168.120.1 sky.wdc.us.gl3 A +short
dig @192.168.120.2 sky.wdc.us.gl3 A +short

# 2. Check DHCP failover peer status
sudo journalctl -u dhcpd | grep -i failover | tail -10

# 3. Simulate SKY unavailability by stopping named and dhcpd on SKY
# (Coordinate with Director of Cyber Security — brief outage for DNS/DHCP)
sudo systemctl stop named dhcpd

# 4. Verify RAIN assumes DNS resolution within 30 seconds
sleep 30
dig @192.168.120.2 sky.wdc.us.gl3 A +short

# 5. Verify DHCP leases are being issued by RAIN
sudo journalctl -u dhcpd -f --since "1 minute ago"

# 6. Restore SKY services
sudo systemctl start named dhcpd
sudo rndc reload

# 7. Verify zone transfer from RAIN back to SKY
sudo rndc sync wdc.us.gl3
dig @192.168.120.1 sky.wdc.us.gl3 A +short

# 8. Log the drill
echo "$(date) [DRILL] DNS failover drill SKY→RAIN completed — RTO verified" >> /var/log/asset-inventory.log

Pass criteria

  • RAIN continues resolving DNS within 30 seconds of SKY unavailability
  • DHCP failover peer activates without client impact
  • SKY rejoins as primary after restart without manual intervention
  • Zone transfer completes and both servers in sync

Drill 4 — auditd immutable mode verification

Objective: Verify auditd is in immutable mode and audit rules cannot be modified at runtime.

# 1. Verify immutable mode is set
sudo auditctl -s | grep "enabled 2"

# 2. Attempt to clear rules — should fail
sudo auditctl -D 2>&1 | grep -i "Operation not permitted\|immutable"

# 3. Verify rules are loaded
sudo auditctl -l | head -20

# 4. Check audit log is being written
sudo tail -10 /var/log/audit/audit.log

# 5. Verify log shipping to WIND ELK
sudo ausearch -ts today -m USER_LOGIN 2>/dev/null | tail -5

Pass criteria

  • auditctl -s shows enabled 2
  • auditctl -D returns "Operation not permitted" or similar
  • Audit log has recent entries
  • Logs visible in Elasticsearch on WIND

Drill schedule

Drill Frequency Next Run Owner
Brute force detection Monthly Q2 2026 IT Admin
AIDE integrity detection Monthly Q2 2026 IT Admin
DNS failover Quarterly Q2 2026 DNS Admin
auditd immutable Quarterly Q2 2026 IT Admin