ai-evals

AI Evaluation and Fine-Tuning Methodology Skill

Core stance: an eval is an instrument. An untrusted instrument is worse than no instrument, because it produces confident wrong numbers that ship regressions. Fine-tuning is an optimization loop around that instrument. If the instrument is weak, training just makes the model better at gaming bad measurement. This skill is the cross-domain methodology layer that domain eval and model-lifecycle skills defer to: how to keep an LLM-as-judge honest, integrate eval frameworks, choose between prompting/context/tools/test-time compute/SFT/preference/RFT/PEFT/ distillation, derive thresholds instead of guessing them, and stop flaky runs or training leakage from masquerading as progress.

This is the methodology umbrella for evals. Domain skills own what to measure; this skill owns whether you can trust the measurement.

• Building an eval system for a coding agent -> ai-coding-agents-observability-evals
• Evaluating RAG / retrieval / search -> ai-rag
• Running Hub model benchmarks (inspect-ai, lighteval) -> use the huggingface-skills: plugin (external)
• General LLM lifecycle decisions -> ai-llm
• This skill: judge bias, framework choice, calibration, reproducibility, optimization technique gates — the parts those four share and none owns in depth.

ASCII Flow

eval need
  |
  v
define verifiable goal  (what would FAIL if the requirement reverted?)
  |
  v
choose grader
  deterministic check  ->  LLM-as-judge  ->  human label  (cheapest that works)
  |
  v
control judge bias
  position / length / self-preference / verbosity
  |
  v
derive thresholds from a labeled calibration set  (not vibes)
  |
  v
choose optimization path  (prompt/RAG/tools -> SFT -> preference/RFT/PEFT)
  |
  v
choose inference-time lift  (self-consistency / rerank / verify / refine)
  |
  v
control flake  (pass@k, low temp, quarantine unstable cases)
  |
  v
trustworthy gate  ->  train / block / ship / rollback