troubleshoot-cockroachdb

Troubleshoot Cockroachdb

When to use this skill

• LSM Compaction Death Spiral: Writes outpace compaction. L0 sublevel count climbs. Read latency goes exponential. Write stalls follow. The node appears "stuck."
• Raft Liveness Failure: A node becomes slow (GC pause, disk stall, CPU saturation), can't process Raft heartbeats, loses leadership, the cluster redistributes leases causing cascading unavailability windows.
• Clock Skew Crisis: NTP misconfiguration causes clock drift. Node self-terminates at >80% of max-offset. If multiple nodes drift (common with shared NTP), quorum is lost.
• Hot Range: A single range receives disproportionate traffic due to sequential key patterns. One node bottlenecks the entire workload while others idle.
• Intent Accumulation: Long-running or abandoned transactions leave write intents that block other transactions, cascading latency across the cluster.
• Memory Pressure / GC Thrashing: Large queries exhaust SQL memory budget. Go GC pauses exceed Raft heartbeat interval (3s), causing liveness loss and oscillating availability.
• Any time the user reports a Cockroachdb service behaving outside its expected envelope (elevated errors, latency, saturation, resource exhaustion, or unexpected restarts).
• An on-call engineer is paging on a Netdata alert tied to a Cockroachdb instance and wants a structured triage path.

Key facts

• This skill wraps the Netdata operator playbook for Cockroachdb. It does not replace the playbook; it routes a coding agent through MCP queries against the same signals the playbook relies on.
• CockroachDB is a distributed, strongly-consistent SQL database built on a replicated key-value store. It layers SQL execution on top of a transactional KV engine that uses Raft consensus for replication and MVCC for concurrency control. To reason about its failures, you must hold several interacting subsystems in your head simultaneously.
• Dominant failure archetypes the playbook calls out: LSM Compaction Death Spiral; Raft Liveness Failure; Clock Skew Crisis; Hot Range; Intent Accumulation.
• Netdata observes the signals listed in the rule files via its native collectors, plus any OpenTelemetry-shipped metrics that your Cockroachdb instrumentation adds. Both paths end at the same MCP query surface.
• Netdata's cockroachdb collector emits 60 context(s) under cockroachdb.*. The rule files enumerate which contexts surface which domain; the Verification section below names the load-bearing ones explicitly.