infrastructure-management

Infrastructure Management Skill

Overview

Manages physical and SBC infrastructure with awareness of hardware boot times, access hierarchies, and out-of-band management capabilities.

Access Strategy — Try in Order

1. SSH key-based — Primary access method
2. SSH password — Fallback if key fails
3. Tailscale/VPN — If direct SSH unreachable
4. OOB management (iDRAC/iLO/IPMI) — For power cycling, console when SSH down
5. Serial console — Last remote option
6. Physical access — Keyboard/monitor as final resort

Always maintain at least two working access methods per host.

Phase 1: Connectivity Audit

For each managed host, check access:

# SSH connectivity check (2s timeout)
ssh -o ConnectTimeout=2 -o BatchMode=yes USER@HOST "echo ok" 2>&1

# Check SSH via password (if key fails)
# NOTE: sshpass usage requires explicit user approval

# Check if OOB/iDRAC is reachable
curl -sk --connect-timeout 5 https://IDRAC_IP/data?get=pwState 2>&1 || echo "iDRAC unreachable"

# IPMI ping check
ipmitool -I lanplus -H IPMI_IP -U root -P PASSWORD power status 2>&1

Report format:

Host: hostname (IP)
  SSH Key:     OK | FAIL (reason)
  SSH Pass:    OK | FAIL | NOT_TESTED
  OOB:         OK (iDRAC6/iLO4/IPMI) | UNREACHABLE
  Boot Time:   ~30s (SBC) | ~3min (desktop) | ~10min (enterprise)
  Status:      HEALTHY | DEGRADED | UNREACHABLE

Phase 2: Hardware Health

Query available health data from each host:

# Temperature (via SSH)
ssh HOST "cat /sys/class/thermal/thermal_zone*/temp 2>/dev/null || sensors 2>/dev/null"

# Disk health
ssh HOST "df -h && smartctl -a /dev/sda 2>/dev/null | grep -E 'Health|Temperature|Reallocated'"

# Memory
ssh HOST "free -h"

# Uptime and load
ssh HOST "uptime"

# Docker status (if applicable)
ssh HOST "docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}' 2>/dev/null"

For iDRAC-equipped servers:

# Sensor readings via REST API
curl -sk --cookie "session_cookie" "https://IDRAC_IP/data?get=tempprobes"
curl -sk --cookie "session_cookie" "https://IDRAC_IP/data?get=fanstatus"

# System Event Log
ssh IDRAC_IP "racadm getsel" 2>/dev/null