benchmark-table-parsing-and-aggregation

benchmark-table-parsing-and-aggregation

Summary

Extract method-by-dataset performance matrices from peer-reviewed benchmarking studies, compute summary statistics (mean, median, rank), and compile results into a standardized TSV format to enable quantitative comparison of competing computational methods. This skill is essential when reproducing or validating claims that one method outperforms another across multiple datasets or evaluation metrics.

When to use

You are reproducing a comparative benchmarking claim (e.g., 'Method A outperforms Method B') from a published study and need to verify the finding by assembling the raw accuracy metrics (NMI, ARI, purity scores, clustering accuracy) reported in the paper's tables or supplementary data, especially when the claim rests on aggregated performance across multiple datasets.

When NOT to use

• The benchmark study reports only qualitative rankings (e.g., 'Method A is better') without numeric accuracy metrics — aggregation requires numeric data.
• The target study is a review or opinion piece, not a direct empirical comparison with raw metric tables.
• You are comparing methods from different papers using different datasets — inter-paper aggregation introduces confounding factors (dataset-specific difficulty, metric definitions) and should be avoided without explicit cross-study normalization.

Inputs

• Peer-reviewed benchmarking paper (preprint or published) with comparative method results
• Supplementary data tables, figures, or SI appendices containing raw or summary accuracy metrics
• Method name list (e.g., chromVAR variants, SnapATAC, baseline methods)
• Dataset identifier list or accession numbers used in the benchmark

Outputs

• Method-by-dataset matrix (TSV) with methods as rows, datasets as columns, metric values as cells
• Summary statistics table (TSV) with rows as methods and columns as mean, median, std dev, rank of performance
• Ranked method list (TSV or CSV) ordered by aggregated performance metric