Skip to content
Scintria
ToolsGrant writerFunding
HomeLiteratureClinical trialsEvidence LabProtocol packData analysisLearnProjectsLibraryReferencesApplications
Start free
ScintriaStart free

Learn / Biostatistics / Hypothesis testing

Rank tests, multiple comparisons and the misuse of p-values

14 minute read · 6-question quiz

Download this lesson as a colour Word or PDF study guide with the figures and answers. Included in paid plans; reading and quizzes stay free.Sign in

What you will learn

  • Apply the logic of rank-based (non-parametric) tests and know when to use them

  • Explain why multiple testing inflates false positives and apply a Bonferroni correction

  • Recognise p-hacking, selective reporting and the problems with the 'p < 0.05' threshold

Rank-based tests: when the mean is the wrong summary

Tests like the t-test rely on means and normality. When data are strongly skewed, ordinal (pain scores), or contain extreme outliers, tests that use only the ranks of the values are safer. They ask whether values in one group tend to be higher than in the other.

QuestionParametric testRank-based alternative
Two independent groupsIndependent t-testMann-Whitney U (Wilcoxon rank-sum)
Paired dataPaired t-testWilcoxon signed-rank
3 or more groupsOne-way ANOVAKruskal-Wallis
Association of two variablesPearson correlationSpearman correlation

How Mann-Whitney works

Pool all the observations, rank them from smallest to largest, then compare the sum of ranks in each group. If groups are alike, high and low ranks are mixed evenly. If one group has mostly high ranks, the rank sum is extreme.

Worked example: pain scores

Seven patients receive a standard analgesic (A) and seven a new one (B). Pain scores (0 to 10) at 1 hour:

  • A: 3, 5, 4, 6, 2, 7, 5
  • B: 8, 9, 6, 10, 7, 9, 10

Almost every B value sits above almost every A value. The Mann-Whitney statistic is U = 2 (counting ties as half, A beats B in only 2 of the 49 possible pairs), and the two-sided p-value is about 0.005. We conclude pain scores tend to be higher with B. The result concerns the whole distribution (the median and more), so report medians (A: 5, B: 9) and IQRs rather than means.

Rank tests are not a cure-all

They lose a little power when data are truly normal (about 5%), and they test a slightly different hypothesis. Their p-values do not by themselves give an estimate of 'how much'; supplement them with a median difference and a bootstrap or Hodges-Lehmann interval.

The multiple comparisons problem

Every test carries a 5% risk of a false alarm. Run many, and false alarms accumulate. With k independent tests at α = 0.05:

P(at least one false positive) = 1 - 0.95^k

  • 5 tests: 22.6%
  • 10 tests: 40.1%
  • 20 tests: 64.2%
0204060801235810152030Number of independent tests (each at α = 0.05)Probability (%)False-positive chance
Testing many outcomes, subgroups or time points: false positives become likely.

Controlling the damage

  • Bonferroni correction: divide α by the number of tests. With 10 tests, each must reach p < 0.05/10 = 0.005. Simple and conservative. (Equivalently, multiply each p-value by 10.)
  • Holm is a uniformly less conservative step-down version of Bonferroni and is a sensible default.
  • False discovery rate (Benjamini-Hochberg) controls the expected share of false findings among those declared significant. Common in genomics and other exploratory work with thousands of tests.
  • Pre-specification: the best protection is to declare one primary outcome, a small number of secondary outcomes and a limited list of subgroups in the protocol, and to report all of them.

Subgroup analyses

Slicing a trial by sex, age, site, severity and baseline marker produces many tests. A result that is 'significant in women over 60' is probably chance unless it was pre-specified, supported by a significant interaction test, and consistent with biology. A famous demonstration: a trial analysis by astrological sign found that aspirin appeared ineffective for Geminis and Libras, an absurd subgroup result produced by chance alone.

P-hacking and the replication crisis

P-hacking means trying analyses until one produces p < 0.05: dropping outliers selectively, switching outcomes, adding covariates, stopping data collection when p dips below 0.05, or testing many exposures. Even without bad intent, flexible analysis ('the garden of forking paths') inflates false positives. Publication bias compounds it: significant results are published and null results are not.

The American Statistical Association's 2016 statement set out principles: p-values do not measure the probability a hypothesis is true; conclusions should not rest on whether p crosses a threshold; proper inference requires full reporting and transparency; and a p-value does not measure the size or importance of an effect.

An honest-analysis checklist

  1. Write a statistical analysis plan before unblinding or looking at outcomes.
  2. Name one primary outcome and the test for it.
  3. Report all outcomes and analyses done, including the null ones.
  4. Report estimates with CIs, not only p-values.
  5. Label anything unplanned as exploratory, and treat it as hypothesis-generating.
  6. Register trials (ISRCTN, ClinicalTrials.gov, PACTR) before the first participant is enrolled.

Summary

  • Rank tests (Mann-Whitney, Wilcoxon, Kruskal-Wallis, Spearman) suit skewed, ordinal or outlier-prone data.

  • With k independent tests the chance of at least one false positive is 1 − 0.95^k.

  • Bonferroni divides α by k; Holm is less conservative; FDR methods suit exploratory screening.

  • P-hacking and selective reporting inflate false positives; pre-registration and an analysis plan prevent them.

  • A p-value is not the size, importance or probability of truth of a finding.

Check your understanding

Question 1
1

Which test is the rank-based alternative to the independent t-test?

Question 2
2

Twenty independent outcomes are each tested at 0.05 with no true effects. The chance of at least one 'significant' result is about:

Question 3
3

With 8 planned comparisons, the Bonferroni-adjusted significance threshold is:

Question 4
4

Pain scores (0 to 10) in 12 patients are strongly skewed. Which summary is best?

Question 5
5

Which is an example of p-hacking?

Question 6
6

A subgroup finding is more credible when it is:

Draft study guide. Spotted an error or something unclear? Tell us at hello@scintria.com. This is educational material, not clinical or regulatory advice.

Comparing means: the t-testsRisk, odds and the measures of association