method-comparison-statistical-summarization

Method-comparison statistical summarization

Summary

Compile accuracy metrics across multiple methods and datasets into a structured matrix, then compute summary statistics (mean, median, rank) to quantitatively rank method performance. This skill transforms scattered benchmark results into a standardized comparison table suitable for identifying superior approaches.

When to use

You have extracted clustering or classification accuracy metrics (NMI, ARI, purity scores) for two or more competing methods evaluated on multiple datasets, and need to determine which method performs best overall rather than on individual datasets alone. This is especially useful when one method outperforms others on some datasets but not others, requiring aggregation to resolve the ranking.

When NOT to use

• Input is a single dataset: summary statistics across datasets are meaningless; report per-dataset results only.
• Methods have not been evaluated on a common set of datasets: direct comparison is invalid; stratify by dataset subset.
• Metrics are on different scales (e.g., one method reports ARI, another reports NMI): standardize or report separately before aggregation.

Inputs

• Clustering accuracy metrics table (from published benchmark or supplementary data)
• List of methods being compared (e.g., chromVAR variants, SnapATAC)
• List of datasets used in benchmarking
• Individual metric values per method-dataset pair (e.g., NMI, ARI, purity scores)

Outputs

• Method-by-dataset TSV table (rows=methods, columns=datasets, cells=accuracy scores)
• Summary statistics table (method, mean accuracy, median accuracy, rank)
• Ranking of methods by overall performance (best to worst)

How to apply

Extract accuracy scores for each method-dataset combination from published benchmark tables or supplementary data (e.g., NMI, ARI, or purity values). Organize these into a matrix with rows representing methods and columns representing datasets. For each method, calculate mean, median, and rank across all datasets. The rationale is that single-dataset comparisons can be dataset-specific; aggregating across multiple datasets using robust summary statistics (median is preferred over mean for outlier robustness) reveals which method generalizes best. Rank methods by their summary statistics to identify the overall winner while documenting which datasets favor which methods, revealing method-dataset interactions.