evaluation-benchmarking

Evaluation And Benchmarking

Ground Rules

• Built-in eval suites are deterministic contract harnesses, not physical-fidelity or media-quality evidence.
• Benchmark direct provider operations only: predict, embed, score, and policy.
• plan() is a WorldForge facade workflow. Do not route benchmark score or policy through it.
• Preserve BenchmarkBudget non-zero exit behavior on violations.
• Preserve claim-boundary and metric-semantics metadata in JSON, Markdown, and CSV renderers.
• Treat benchmark/eval output as evidence only when the fixture, command, provider surface, and renderer semantics are all current and inspectable.

Workflow

1. Read src/worldforge/evaluation/suites.py for eval changes or src/worldforge/benchmark.py for benchmark changes.
2. Validate inputs eagerly through BenchmarkInputs and load_benchmark_inputs(...); reject unknown keys and non-finite metrics.
3. Keep examples/benchmark-inputs.json and examples/benchmark-budget.json reproducible and checkout-safe.
4. Keep score and policy payloads JSON-native; for provider-native tensors or arrays, preview type and shape rather than forcing raw encoding.
5. If operation surfaces or CLI text change, update help snapshots, harness diagnostics, README, docs/src/benchmarking.md, docs/src/api/python.md, docs/src/playbooks.md, and changelog together.
6. Test direct operation behavior, input parsing, budget pass/fail paths, and renderer output.

Definition Of Done

• The changed suite or benchmark path has focused tests for success and failure behavior.
• Rendered artifacts remain JSON-native and internally coherent before Markdown/CSV output.
• Claims in README/docs/changelog match the actual command and provider used.
• The final validation includes the focused tests plus docs/provider checks when public text changed.

Metric Semantics

• Latency is process-local wall-clock timing for successful samples.
• Retry counts come from emitted ProviderEvent records.
• Throughput is successful samples over elapsed time.
• Event rows can aggregate attempts; sum request_count when reporting actual request volume.