kanban-dispatcher-hardening

2. worker watchdog (Phase 0 — dead worker 즉시 reclaim)

문제

기존 dispatcher는 TTL 만료(claim_expires < now)시에만 in_progress task를 reclaim. 그런데 worker process가 비정상 종료 (segfault, OOM kill, parent kill)되면 claim_expires가 만료될 때까지 1시간 동안 queue stuck.

해결

Phase 0 추가: os.kill(pid, 0)로 worker 생존 확인, dead면 즉시 reclaim.

Board 필터 적용 (cross-board race 차단): 각 dispatcher는 자기 board의 in_progress만 본다. default/content/integrations 3개 dispatcher가 동시에 돌 때, 한 dispatcher가 다른 board의 dead worker를 reclaim해버리는 race를 차단.

# Phase 0: watchdog (board-scoped)
in_progress = conn.execute('''
    SELECT id, title, worker_pid FROM tasks
    WHERE status = "in_progress" AND worker_pid IS NOT NULL AND board = "self_board"
''').fetchall()

content는 legacy 호환:

WHERE status = "in_progress" AND worker_pid IS NOT NULL AND (board = "content" OR board = "" OR board IS NULL)

Phase 1 (TTL reclaim)도 동일하게 board 필터:

stale = conn.execute('''
    SELECT id, title, worker_pid, claim_expires FROM tasks
    WHERE status = "in_progress"
      AND claim_expires IS NOT NULL
      AND claim_expires < ?
      AND board = "self_board"
    ORDER BY claim_expires ASC
''', (now_ts,)).fetchall()

# Phase 0: watchdog
in_progress = conn.execute('''
    SELECT id, title, worker_pid FROM tasks
    WHERE status = "in_progress" AND worker_pid IS NOT NULL
''').fetchall()

for task_id, title, wpid in in_progress:
    try:
        os.kill(int(wpid), 0)  # signal 0 = existence check (no actual signal)
    except (ProcessLookupError, OSError):
        # Worker dead — 즉시 reclaim
        conn.execute('UPDATE tasks SET status="ready", worker_pid=NULL, claim_expires=NULL WHERE id=?', (task_id,))
        result['watchdog_reclaimed'] += 1

출력 형식

watchdog_reclaimed=N | ttl_reclaimed=M | claimed=K | spawned=L

• watchdog_reclaimed: Phase 0가 dead worker로 reclaim한 수
• ttl_reclaimed: Phase 1가 TTL 만료로 reclaim한 수 (기존)
• claimed: Phase 2가 ready task를 claim한 수
• spawned: Phase 3가 worker를 spawn한 수