statistical-conventions

Statistical conventions

The design and the measurement scale choose the test, not the size of the p-value it yields. Every reported number travels with the quantity it estimates, its uncertainty, the assumption it rests on, and the software that computed it.

Choosing the test

1. Design first. The same units measured twice (before and after, matched pairs, two methods on the same samples) are paired: scipy.stats.ttest_rel, wilcoxon. Independent groups: ttest_ind(a, b, equal_var=False) (Welch; use Student's only when equal variances are a design fact, and say so), mannwhitneyu. Three or more groups: f_oneway or kruskal; repeated measures across blocks: friedmanchisquare. Name the unit of analysis before choosing: observations nested in a unit (cells within a donor, trials within a participant, technical replicates within a sample, regions within a patient) are aggregated to that unit (a per-donor proportion or mean) or fitted with a mixed model before any test. Testing the nested observations as if independent inflates n by orders of magnitude and makes every p-value meaningless.
2. Scale and distribution. Ordinal outcomes, heavy skew, small n with outliers, or censored-at-a-bound measurements call for rank-based tests (Mann-Whitney, Wilcoxon signed-rank, Kruskal-Wallis, Spearman spearmanr, Kendall kendalltau). Continuous, roughly symmetric data with adequate n use the t family. Check rather than assume: shapiro (informative for modest n only), a Q-Q plot, levene for variances. Name what you looked at; a normality test on n = 5 or n = 5000 says little on its own. Intensity measurements (proteomics and metabolomics signal, microarray and normalized expression values, fluorescence, concentrations spanning decades) are compared on the log2 scale: the difference of log2 means is the fold-change the question asks about, and a t-test on the linear values tests the wrong quantity. "Normalized", "imputed" or "batch-corrected" does not mean "log-transformed": values in the thousands or millions are linear, and the transform is part of the prescribed workflow even when the instruction does not spell it out. The scale is decided per step, not once for the pipeline: the statistics you compute (means, fold-changes, t-tests, correlations) take log2; a tool that is handed the expression matrix (GSEA, clustering, a named pipeline) takes the columns the task names, as supplied, unless the task or the tool's own documentation says to transform them. When the two readings give different calls, run both, say which the task's wording supports, and report the other as a sensitivity. Say which scale each statistic was computed on.
3. Ordered categories need a trend test, not an omnibus test. Doses, grades, stages, exposure levels: Jonckheere-Terpstra for a continuous or ordinal outcome across ordered groups (R clinfun::jonckheere.test, PMCMRplus::jonckheereTest, DescTools::JonckheereTerpstraTest); Cochran-Armitage for a binary outcome across ordered groups (R stats::prop.trend.test, DescTools::CochranArmitageTest; Python statsmodels.stats.contingency_tables.Table(counts).test_ordinal_association(), whose default scores give Cochran-Armitage); Page's L for ordered treatments in a blocked design (scipy.stats.page_trend_test). A Kruskal-Wallis p says "some group differs"; it is not evidence of a monotone trend. State the scores assigned to the levels.
4. Counts and proportions. chi2_contingency (state whether Yates' correction was applied), fisher_exact when any expected count is below five, barnard_exact or boschloo_exact as more powerful exact alternatives. Odds ratio with interval: statsmodels.stats.contingency_tables.Table2x2(t).oddsratio_confint(); risk ratio via riskratio_confint(). Report the 2x2 table itself, not only the ratio.
5. Correlation. Pearson for a linear relation between roughly bivariate-normal variables, Spearman otherwise; give r with its interval (scipy.stats.pearsonr(x, y).confidence_interval()) and n. Correlation on aggregated means is not correlation on individuals; say which level the r describes.