recover

linstor:recover

Specialized skill for diagnosing and recovering DRBD/LINSTOR storage issues in Kubernetes clusters.

When to use

• linstor r l --faulty shows broken resources
• DRBD resources in StandAlone, Connecting, Inconsistent, Outdated, or Unknown state
• Stuck DELETING resources that won't complete
• TCP port collisions between DRBD devices
• Stuck toggle-disk operations
• Nodes with drbd.linbit.com/lost-quorum taints
• drbdadm adjust failing with bitmap or connection errors

Core Principles

1. Match the operator's natural language. Detect from prior conversation messages. Use that language in every prompt, AskUserQuestion option, summary, and gate. Never ask "what language?" separately. Code identifiers, linstor / drbdadm commands, file paths, and GitHub-public text stay in their canonical form (usually English). 0a. One valid path → just do it. When the diagnostic graph picks one resource to fix and one operation that's not destructive (re-attach, re-promote, refresh), the skill runs it without "ok to do this?" friction. The dangerous-operations gate at principle 8 still applies — linstor node lost, deleting the last replica, drbdadm down on InUse/Primary, --discard-my-data with one diskful copy, drbdadm create-md --force — those always ask. Safe single-path operations don't. 0b. Front-load the interview. Read the full diagnostic graph first (linstor r l --faulty, linstor n l, drbdadm status on each affected node, linstor error-reports list), then present a single recovery plan with the ordered operations, classifications (safe / dangerous), and the dangerous-operation approvals batched into one approval screen. Operator either approves the lot or names what to skip. Phases of the recovery run uninterrupted against the collected approvals. 0c. Layer-pure operator output. The skill never says "returning control to wizard" or makes any orchestration commentary in the operator-facing summary. linstor:recover is a standalone skill that nobody auto-dispatches today, but even if it ever gets called from a chain, whoever invoked it figures out what's next on their own.
2. Work one resource at a time. On mass incidents, resist the urge to fix everything at once. Serial, monotonic recovery is safer.
3. Always verify on the node itself. LINSTOR's view can be stale or wrong. drbdadm status on the satellite is the source of truth.
4. Preserve UpToDate replicas. Never touch the source-of-truth replica first. Fix broken copies by working outward from the healthy one.
5. ALWAYS verify at least one UpToDate diskful replica exists before ANY destructive action. Run linstor r l -r <resource> and confirm there is at least one replica with State=UpToDate and a real storage pool (not DfltDisklessStorPool). If there are zero UpToDate diskful replicas — STOP and ask the user before proceeding. This is the single most important safety check.
6. If SyncTarget is progressing, stop. A resource that entered sync is already recovering. Don't interfere.
7. Every action should simplify the graph. Remove peers from conflict, reduce topology complexity. If an action creates new noise, don't do it.
8. Check error-reports for the real cause. What looks like StandAlone may actually be an adjust failure underneath.
9. ASK the user before dangerous operations. Always ask for explicit confirmation before: linstor node lost, deleting the last replica, drbdadm down on InUse/Primary resource, --discard-my-data when only one diskful copy remains, drbdadm create-md --force (wipes DRBD metadata and triggers a full resync).