long-eval

Long (Restart-Durable) Eval

Role: worker + supervisor. Runs a checkpointed eval that resumes across container recycles, and keeps it alive until it finishes.

Why this exists

A cloud session runs in a container that is recycled on a short cadence (often every few minutes). A recycle kills every running process but keeps the working tree. So any eval longer than one container lifetime must:

1. Checkpoint to disk at fine granularity, and resume from the checkpoint rather than restarting — otherwise every recycle throws the work away.
2. Be relaunched after each recycle. The session itself is the only supervisor: each recycle re-invokes it ("Continue from where you left off"), which is the relaunch trigger — tighter than any timer.

This skill ships both halves so you don't rebuild them:

File	Role
durable_runner.py	The engine — grades one cell at a time, checkpoints after every sample, resumes at the next unbanked sample. Import it or run via the CLI.
run_eval.py	CLI supervisor — run, status, and ensure-alive (the guard-relaunch primitive).

${CLAUDE_PLUGIN_ROOT} below is the plugin root; from this repo it is plugins/dev-team.

The eval module contract

An eval module is a plain Python file supplying only the eval-specific bits. Everything durable is handled for you.

SAMPLES = 5           # optional: trials per cell (majority vote)
WORKERS = 10          # optional: moderate on purpose — see note below

def cells():          # -> list of (key, payload)
    ...               # key = a STABLE string, identical every launch
def sample(payload):  # -> json-serializable; ONE independent trial
    ...
def reduce(samples, key, payload):  # optional -> the cell's record dict
    ...
def finalize(done, out_dir):        # optional -> summary (once all cells bank)
    ...
def flag(record):                   # optional -> bool: force a progress line
    ...

key is the checkpoint identity — it must be byte-identical across relaunches (e.g. f"{fixture}||{variant}"). payload is passed straight to sample and is never persisted, so cells() reconstructs it each launch.