codec-rd-eval

This skill automates the full Rate-Distortion evaluation pipeline:

1. Build encoder binaries from refc/*.c (C11, gcc)
2. Run parallel encoding simulations across (sequence, QP, config) combinations
3. Compute BD-PSNR/BD-rate using VCEG-M33 polynomial interpolation (3+ points)
4. Generate comparison reports with per-sequence and aggregate metrics

Scope: Encoder RD evaluation only. This skill evaluates encoder quality metrics (BD-PSNR, BD-rate, optional SSIM/VMAF). For decoder conformance testing against JVET/JCTVC bitstreams, use /rtl-agent-team:codec-conformance-eval.

Phase-agnostic: While commonly used during rat-dse Step 3b, this skill can be invoked at any Phase where quantitative RD comparison of encoder configurations is needed — Phase 1 (algorithm exploration), Phase 2 (architecture validation), Phase 4 (fixed-point precision impact), or standalone evaluation outside the pipeline.

Execution modes:

• local: ProcessPoolExecutor-based parallel encoding on local CPU cores
• aws-batch: Optional AWS Batch spot instance submission for large-scale evaluation

Key features:

• N-candidate comparison: Compare 2+ configurations via candidates[] array (anchor + N tests)
• Configurable encoder CLI: encoder_cmd_template supports any encoder (HM, VTM, custom). Available variables: {encoder}, {cfg}, {input}, {width}, {height}, {fps}, {frames}, {qp}, {bitstream}, {recon}, {bit_depth}, {chroma_format}
• Configurable output parsing: Custom regex patterns for bitrate/PSNR extraction
• SSIM/VMAF opt-in: Additional quality metrics on explicit user request only
• bit_depth/chroma_format aware: YUV weighting adjusts per chroma format (420/422/444)
• 3+ QP point support: Standard 4-point (exact fit), 5+ (least-squares), 3 (quadratic fallback)

<Use_When>

• Comparing algorithm candidates with objective quality metrics (any Phase)
• Measuring BD-PSNR/BD-rate between anchor and modified encoder configurations
• Evaluating fixed-point precision impact on codec quality (e.g., 12-bit vs 16-bit paths)
• Validating that HW-friendly algorithm modifications preserve acceptable quality
• N-way comparison of multiple encoder configurations (candidates[] mode)
• The user explicitly says "RD eval", "BD-PSNR", "BD-rate", "codec quality", "algorithm quality evaluation" </Use_When>

<Do_Not_Use_When>

• No ref C model encoder exists yet (build ref model first via ref-model skill)
• Comparing RTL vs C model output (use /rtl-agent-team:rtl-model-consistency instead)
• Running decoder conformance tests against standard bitstreams (use /rtl-agent-team:codec-conformance-eval)
• Running RTL-level conformance against reference decoder (use /rtl-agent-team:rtl-conformance-test)
• Non-codec designs where RD metrics don't apply </Do_Not_Use_When>

<Why_This_Exists> In codec design, algorithm selection has the highest impact on final quality and area. Theoretical complexity analysis (operations/pixel, gate estimates) provides useful guidance but cannot capture the full picture — actual RD performance on representative sequences is the definitive metric.

BD-PSNR/BD-rate (VCEG-M33) is the universally accepted method in the video coding community for comparing codec configurations. It normalizes across different operating points (QP values) to produce a single, meaningful comparison metric.

Without this skill, teams either skip quantitative RD evaluation (risking suboptimal algorithm selection) or manually set up evaluation infrastructure (time-consuming and error-prone). </Why_This_Exists>

<Execution_Policy>

• Requires refc/*.c to exist (or user-specified encoder source path)
• HJSON test configuration defines all evaluation parameters
• Local execution is the default; AWS Batch is opt-in via configuration
• Simulation results are cached at .rat/scratch/rd-eval/ for re-analysis
• Report is generated at the path specified in test configuration (default: docs/phase-1-research/rd-eval-report.md)
• On build failure: report error details and stop (do not proceed with stale binaries)
• On simulation failure: report failed jobs, compute BD metrics from successful jobs with warnings
• On metric parsing failure (bitrate=0 or PSNR=0): mark job as failed with guidance to check output_parsing
• timeout_per_job is in seconds (default: 3600s = 1 hour per encoding job)
• SSIM/VMAF are computed ONLY when explicitly requested via quality_metrics config
• Dependencies: gcc (C11), Python 3.9+, numpy, hjson. Optional: ffmpeg (required for SSIM/VMAF computation when quality_metrics includes "ssim" or "vmaf"), boto3 (required only for aws-batch execution mode)
• Self-test: python3 {plugin_root}/skills/codec-rd-eval/scripts/bd_rate.py --test runs built-in unit tests ({plugin_root} = plugin root resolved from .rat/state/spawn-context.json) </Execution_Policy>