aiter-op-test

aiter op_test standard

How every aiter op test in op_tests/test_*.py must be built. The canonical reference in-tree is op_tests/test_quant.py — match its shape. A test is both a correctness check (vs a torch reference) and a perf sweep that ends in a markdown summary table.

Follow this whenever you create a new test_*.py, rewrite an old one, or add shapes/candidates to an existing one.

The hard rules

1. Mirror test_quant.py. Same imports, same decorator, same table-at-the-end flow. Don't invent a different structure.
2. @benchmark() on the test fn. It logs the function's call args (the shape params) as table columns automatically and merges the dict you return. So the test fn signature is the table's left-hand columns — name params accordingly.
3. Candidates live in a dict; build ret in a loop. Per candidate record raw us, plus TFLOPS and TB/s, plus err — ret[f"{name} us"], ret[f"{name} TFLOPS"], ret[f"{name} TB/s"], ret[f"{name} err"]. Never hand-write ratio columns. (TFLOPS/TB-s section below.)
4. torch is the reference only — compute it, compare against it, but do not time it and do not put it in the table. (A pure-torch candidate is allowed only when torch is one of the kernels under test, e.g. torch.einsum.)
5. Time with run_perftest, check with checkAllclose — both, for every candidate. Compare in fp32 (.to(dtypes.fp32)).
6. End with a markdown summary table — one per test function. Sweep the shape lists with itertools.product, collect per-shape dicts into a pd.DataFrame, print via aiter.logger.info("... :\n%s", df.to_markdown(index=False)). A file with several test fns of different arg signatures emits one table each — never force-merge them (it scatters NaN columns). Mandatory — a test with no summary table is incomplete.
7. __main__ guard. All argparse + the sweep loop go inside main(), called under if __name__ == "__main__": main(). The reference (run_torch) and the @benchmark test fn stay at module top level so other scripts can import them for combination testing.
8. Standard argparse only. Use -d/--dtype, -b/--batch, -s/--mnk plus op-specific sweep axes as needed (e.g. --layout, --modes, --mtp). Those are legitimate data lists. Do not add bespoke behavior-toggle flags (no --dsv4, no --only-*) — every flag is a list the sweep iterates.
9. Run clean on every supported card. Gate on get_gfx() in main() so the test passes on all supported archs; arch-unsupported ops/candidates are filtered out before launch. Prefer the kernel's arch-dispatching wrapper over a file-per-arch (full section below).