llm-agents-from-scratch: Skill for Claude Code

Benchmark Ollama Cloud models

Pick the default Ollama Cloud model by what the chapter notebooks' agents do, not whether the notebooks finish. The tool is _scripts/benchmark_models.py; this skill is the procedure around it.

Why behaviour, not pass/fail

A notebook can run to completion while the model gets the lesson wrong. Both of these happened with gemma4:31b:

• Faked tool calls: ch04 "passed" while the agent made no tool calls and wrote a trace claiming it had.
• Subagent out of steps: ch09 "passed" because the coordinator handles the failure gracefully.

benchmark_models.py records ran (no errors) and behaved (ran and met every check in its EXPECTATIONS table for that chapter). Rank by behaved.

When to run it

• Before changing the Ollama Cloud default in src/llm_agents_from_scratch/notebook_utils/llm.py (_MODELS["ollama-cloud"]).
• When the current default is retired, or Ollama Cloud adds models.
• Occasionally, to catch drift in the current default.

Procedure

Load the keys first: set -a && source .env && set +a.

1. List the models:

   uv run python _scripts/benchmark_models.py models
   

2. Stage 1: screen every model on the quick chapters (ch03–ch07), plus ch04 two more times, since its failure mode is intermittent. Run models in parallel; these notebooks bind no ports: run appends, so start from a fresh file; otherwise an earlier benchmark's records leak into this ranking:

   rm -f stage1.jsonl
   uv run python _scripts/benchmark_models.py models | xargs -P 3 -I{} \
     uv run python _scripts/benchmark_models.py run {} --out stage1.jsonl
   uv run python _scripts/benchmark_models.py models | xargs -P 3 -I{} \
     uv run python _scripts/benchmark_models.py run {} --chapters ch04 \
       --runs 2 --out stage1.jsonl
   uv run python _scripts/benchmark_models.py summary stage1.jsonl
   

   Keep the models with a perfect stage-1 score, usually a handful.
3. Stage 2: the shortlist, 3 runs of every chapter with a behaviour check, so nothing that decides the default rests on a single run. ch08 is the approval gate; ch09 has concurrent subagents, chaining and human input, and is where models most often fail:

   rm -f stage2.jsonl
   printf '%s\n' <model> <model> ... | xargs -P 3 -I{} \
     uv run python _scripts/benchmark_models.py run {} \
       --chapters ch04,ch05,ch06,ch07,ch08,ch09 --runs 3 --out stage2.jsonl
   uv run python _scripts/benchmark_models.py summary stage2.jsonl
   

   This takes 20–30 minutes; ch09 alone is 2–5 minutes per run.
4. Pick: the highest behaved score, then the fastest (readers wait on ch09). Name a backup, the next model with a perfect score.
5. Check the pick on what the benchmark skips, with OLLAMA_MODEL=<pick> and an isolated HOME (see Pitfalls):
   • ch10 (A2A peer apps; the CrewAI peer uses ollama_chat/<pick>)
   • ch02–ch03
   • one _scripts/run_notebook.py pass over more-examples/ and capstones/