codec-conformance-eval

This skill automates the full decoder conformance evaluation pipeline:

1. Build decoder binary from refc/*.c (C11, gcc)
2. Run parallel decoding of conformance bitstreams (JVET, JCTVC, 3rd party)
3. Compare decoded output against golden references (MD5, bitexact, optional PSNR/SSIM/VMAF)
4. Generate conformance report with profile/level coverage matrix

Scope: Decoder conformance evaluation at the algorithm/C model level.

• This skill tests the C reference model decoder against official conformance streams
• For RTL-level conformance testing, use /rtl-agent-team:rtl-conformance-test (Phase 5)
• For encoder RD evaluation, use /rtl-agent-team:codec-rd-eval

Phase-agnostic: Commonly used during Phase 1-2 (Research/Architecture) to validate decoder algorithm correctness before committing to hardware implementation. Can also be used at any Phase where decoder conformance verification is needed.

	rtl-conformance-test	codec-conformance-eval
Target	RTL implementation (.sv)	C ref model decoder
Phase	Phase 5 (Verify)	Phase 1-2 (Research/Architecture)
Comparison	JM/HM reference output	Official conformance golden output
Purpose	RTL-level bitexact verification	Algorithm-level conformance verification

Execution modes:

• local: ProcessPoolExecutor-based parallel decoding on local CPU cores
• aws-batch: Optional AWS Batch spot instance submission for large stream sets

Key features:

• Profile/level filtering for targeted conformance testing
• Multiple conformance sources: JVET/JCTVC (mandatory per standard), 3rd party (optional)
• Configurable comparison: MD5 checksum, bitexact byte comparison, PSNR threshold
• SSIM/VMAF opt-in (only on explicit user request)
• Profile coverage matrix generation

<Use_When>

• Validating decoder algorithm correctness against official conformance streams
• Verifying that a C ref model decoder passes JVET/JCTVC conformance requirements
• Testing decoder support for specific profiles/levels before hardware implementation
• Evaluating 3rd-party conformance streams for additional coverage
• The user explicitly says "decoder conformance", "conformance stream", "conformance test", "decoder verify", "bitexact decoder" </Use_When>

<Do_Not_Use_When>

• No ref C model decoder exists yet (build ref model first via /rtl-agent-team:ref-model)
• Testing RTL decoder implementation (use /rtl-agent-team:rtl-conformance-test, Phase 5)
• Comparing encoder quality (use /rtl-agent-team:codec-rd-eval)
• Comparing RTL vs C model output (use /rtl-agent-team:rtl-model-consistency) </Do_Not_Use_When>

<Why_This_Exists> Decoder conformance is a hard requirement for any video codec implementation. A decoder that fails official conformance streams has algorithmic bugs that will propagate to hardware.

Catching conformance failures at the C model level (Phase 1-2) is orders of magnitude cheaper than discovering them after RTL implementation (Phase 5). Official conformance streams from JVET and JCTVC exercise specific codec features and corner cases that typical test sequences may not cover.

This skill automates the tedious process of running hundreds of conformance streams, comparing outputs, and tracking which profile features are covered. </Why_This_Exists>

<Execution_Policy>

• Requires refc/*.c with decoder functionality (or configured decoder_src)
• HJSON conformance configuration defines all test parameters
• Local execution is the default; AWS Batch is opt-in via configuration
• AWS Batch decoder contract: the job definition runs /app/decode.sh from a container image you supply, so decoder.decoder_binary does not apply — the image provides the decoder. decoder_cmd_template and extra_args are forwarded as the DECODER_CMD_TEMPLATE and DECODER_EXTRA_ARGS environment variables; an AWS run only matches a local run if the image reads them. The submitter prints a note whenever these settings are set, so the divergence is never silent.
• Profile and level filters (target.profile, target.level) apply identically in local and AWS Batch mode. Level matching accepts 4.1, 4_1 and the packed L41 filename conventions
• Conformance results are cached at .rat/scratch/conformance-eval/
• Report is generated at configured path (default: docs/phase-1-research/conformance-eval-report.md)
• On build failure: report error details and stop
• On decoding failure: mark stream as FAIL, continue with remaining streams
• Mandatory streams (JVET, JCTVC) must all PASS for overall conformance PASS
• Optional streams (3rd party) failures are reported but do not affect overall verdict
• SSIM/VMAF are computed ONLY when explicitly requested via quality_metrics config
• timeout_per_job is in seconds (default: 300s = 5 min per decoding job)
• Dependencies: gcc (C11), Python 3.9+, hjson, numpy (optional, for PSNR computation). Optional: ffmpeg (required for SSIM/VMAF computation when quality_metrics includes "ssim" or "vmaf"), boto3 (required only for aws-batch execution mode)
• Self-test: python3 {plugin_root}/skills/codec-conformance-eval/scripts/compare_output.py --test runs built-in unit tests ({plugin_root} = plugin root resolved from .rat/state/spawn-context.json) </Execution_Policy>