doris-debug-tablet

Tablet & Replica

Quick scan

SHOW PROC '/cluster_health/tablet_health';
SHOW PROC '/statistic';
SHOW PROC '/tasks';

Causes

ID	Cause	Evidence	Source anchor
A	Replica missing / unhealthy	SHOW PROC '/cluster_health/tablet_health' Health≠TOTAL	TabletHealth.java
B	Version skew	tablet version lag > 3 behind quorum	TabletInvertedIndex.cpp
C	Clone backlog	SHOW PROC '/tasks' CLONE queue depth > 50	CloneTask.cpp, BE clone_worker_count
D	Disk skew	SHOW PROC '/statistic' one disk > 85% while others < 50%	DiskBalancer / balance config
E	Single tablet too large	tablet size > compaction_tablet_size_threshold	config.cpp + compaction

10 min triage

-- 1. Global health snapshot
SHOW PROC '/cluster_health/tablet_health';

-- 2. Per-table replica drilldown
ADMIN SHOW REPLICA STATUS FROM db.table WHERE ReplicaStatus != 'OK';

-- 3. Task queue depth
SHOW PROC '/tasks';

# BE-side tablet count & version lag
./scripts/doris-debug be-metrics --be http://$BE:8040 --grep "tablet|version|clone"
./scripts/doris-debug log-grep be/log --pack versions

# Check specific tablet distribution
curl -s "http://$FE:8030/api/tablets_distribution?db=db_name&table=tbl_name"

Cause A — Replica missing

-- Identify unhealthy replicas
ADMIN SHOW REPLICA STATUS FROM db.table WHERE ReplicaStatus != 'OK';

-- Repair with low priority (production safe)
ADMIN REPAIR TABLE db.table PRIORITY = 'LOW';

-- For urgent single-tablet repair
ADMIN REPAIR TABLE db.table PARTITION (p202401) PRIORITY = 'HIGH';

Check BE heartbeat_service_mgr.cpp — heartbeats from a BE with missing tablets won't self-heal if compaction or clone threads are exhausted.

Cause C — Clone backlog

Clone concurrency: clone_worker_count (default 3 in config.cpp). A high Clone backlog often means:

1. Tablet migration triggered by balance_load_disk_safe_threshold / storage_high_watermark_usage_percent
2. Clone threads can't keep up with the repair plan