Learn / Biostatistics / Hypothesis testing
Rank tests, multiple comparisons and the misuse of p-values
14 minute read · 6-question quiz
What you will learn
Apply the logic of rank-based (non-parametric) tests and know when to use them
Explain why multiple testing inflates false positives and apply a Bonferroni correction
Recognise p-hacking, selective reporting and the problems with the 'p < 0.05' threshold
Rank-based tests: when the mean is the wrong summary
Tests like the t-test rely on means and normality. When data are strongly skewed, ordinal (pain scores), or contain extreme outliers, tests that use only the ranks of the values are safer. They ask whether values in one group tend to be higher than in the other.
| Question | Parametric test | Rank-based alternative |
| Two independent groups | Independent t-test | Mann-Whitney U (Wilcoxon rank-sum) |
| Paired data | Paired t-test | Wilcoxon signed-rank |
| 3 or more groups | One-way ANOVA | Kruskal-Wallis |
| Association of two variables | Pearson correlation | Spearman correlation |
How Mann-Whitney works
Pool all the observations, rank them from smallest to largest, then compare the sum of ranks in each group. If groups are alike, high and low ranks are mixed evenly. If one group has mostly high ranks, the rank sum is extreme.
The multiple comparisons problem
Every test carries a 5% risk of a false alarm. Run many, and false alarms accumulate. With k independent tests at α = 0.05:
P(at least one false positive) = 1 - 0.95^k
- 5 tests: 22.6%
- 10 tests: 40.1%
- 20 tests: 64.2%
Controlling the damage
- Bonferroni correction: divide α by the number of tests. With 10 tests, each must reach
p < 0.05/10 = 0.005. Simple and conservative. (Equivalently, multiply each p-value by 10.) - Holm is a uniformly less conservative step-down version of Bonferroni and is a sensible default.
- False discovery rate (Benjamini-Hochberg) controls the expected share of false findings among those declared significant. Common in genomics and other exploratory work with thousands of tests.
- Pre-specification: the best protection is to declare one primary outcome, a small number of secondary outcomes and a limited list of subgroups in the protocol, and to report all of them.
P-hacking and the replication crisis
P-hacking means trying analyses until one produces p < 0.05: dropping outliers selectively, switching outcomes, adding covariates, stopping data collection when p dips below 0.05, or testing many exposures. Even without bad intent, flexible analysis ('the garden of forking paths') inflates false positives. Publication bias compounds it: significant results are published and null results are not.
The American Statistical Association's 2016 statement set out principles: p-values do not measure the probability a hypothesis is true; conclusions should not rest on whether p crosses a threshold; proper inference requires full reporting and transparency; and a p-value does not measure the size or importance of an effect.
Summary
Rank tests (Mann-Whitney, Wilcoxon, Kruskal-Wallis, Spearman) suit skewed, ordinal or outlier-prone data.
With k independent tests the chance of at least one false positive is 1 − 0.95^k.
Bonferroni divides α by k; Holm is less conservative; FDR methods suit exploratory screening.
P-hacking and selective reporting inflate false positives; pre-registration and an analysis plan prevent them.
A p-value is not the size, importance or probability of truth of a finding.
Check your understanding
Draft study guide. Spotted an error or something unclear? Tell us at hello@scintria.com. This is educational material, not clinical or regulatory advice.