safe-update-restart

Safe Update & Restart for Production VPS

🐳 Running in a CONTAINER? Read fleet-update-container instead — several commands here do NOT work there.

If your agent runs from a Docker image (no writable /opt/hermes, no systemd, /opt/data is a bind mount), the commands below are wrong for your box: hermes update refuses in-container, systemctl does not exist (services are s6), and the update is a host recreate, not an agent action. Use the companion skill fleet-update-container for the container-correct procedure — it also covers how to read the real version from the image's install stamp and the boot trap that false-flags lost ports.

When to Use

• Updating Hermes Agent (hermes update) on a box that runs live services.
• Restarting the gateway, x402 gateway, or any service someone may be connected to.
• Any restart where "someone could be connecting any minute."
• Recovering after a service or gateway change.

The Core Principle

Never blind-restart. A production VPS hosts many live APIs, x402 services, and multiple agent gateways that share one Hermes venv. A careless update/restart can drop a live connection or take down a service with no warning. Always: backup → controlled window → restart each service → verify everything after.

The Procedure

Step 1 — Record the baseline (BEFORE touching anything)

Capture current state so you can prove nothing regressed:

# hermes gateways (note PIDs)
for p in <profile1> <profile2> <profile3>; do
  pid=$(pgrep -f "profile $p gateway run" | head -1)
  echo "$p: pid=${pid:-DOWN}"
done
# systemd services
for svc in <x402-api> <x402-backend@...> <other services>; do
  echo "$svc: $(systemctl is-active "$svc" 2>/dev/null)"
done
# public endpoints
for u in "https://api.example.net/" "https://api.example.net/status"; do
  echo "$u -> $(curl -s -m 12 -o /dev/null -w '%{http_code}' "$u")"
done

Step 2 — Take a rollback-safe backup

Use the quick snapshot (covers config, .env, auth, state.db, cron — the rollback-critical files). A full zip of ~/.hermes can be 30-40GB+ and take 30+ min — too heavy for rollback needs.

hermes backup -q -l pre-update-$(date +%Y%m%d) -o /root/hermes-quick-backup.zip

Also record the git rollback point:

cd /usr/local/lib/hermes-agent && git log -1 --oneline && git describe --tags

PITFALL: A full hermes backup (no -q) zips the entire 30-40GB ~/.hermes including cargo/node caches — it's slow and produces giant zips that fill disk. Use -q unless you have a specific reason for the full archive. Delete any stale .partial files it leaves behind.