Learn / Biostatistics / Hypothesis testing
The logic of a significance test
14 minute read · 6-question quiz
What you will learn
State null and alternative hypotheses for a research question
Define the p-value correctly and say what it does not mean
Distinguish type I and type II errors and explain significance level, power and their trade-off
A trial compares systolic blood pressure after 12 weeks on a new regimen with usual care. The groups differ by 6.7 mmHg. Is that a real effect, or could chance alone, in a sample of this size, produce a gap that large? A hypothesis test answers that question in a structured way.
Step by step
- 1
1. State the hypotheses
The null hypothesis (H0) says there is no difference or association: the true mean difference is 0. The alternative hypothesis (H1) says there is. Most tests are two-sided: H1 is simply 'different', in either direction.
- 2
2. Choose the significance level
Before looking at the data, decide how much risk of a false alarm you will tolerate. Convention is α = 0.05.
- 3
3. Calculate a test statistic
Measure how far the observed result is from what H0 predicts, in standard-error units (for example a t or z value).
- 4
4. Find the p-value
The probability of getting a result at least as extreme as the one observed, if H0 were true.
- 5
5. Decide and report
If p < α, reject H0 ('statistically significant'); otherwise 'do not reject' (never 'accept'). Report the estimate and CI, not only p.
Two ways to be wrong
Whatever you decide, there are two mistakes you can make.
- Type I error (false positive, α): concluding there is an effect when there is none. Controlled by the significance level.
- Type II error (false negative, β): missing a real effect. Power = 1 − β is the chance of detecting an effect of a given size. Trials aim for 80% to 90%.
Lowering α (more stringent) reduces false alarms but raises β unless the sample size grows. The cut-off moves right, and more of the real-effect curve falls on the 'miss' side. The only free lunch is a larger sample, a bigger true effect, or less variable measurements.
One-sided or two-sided?
A two-sided test looks for differences in either direction and is the default in clinical research. A one-sided test is justified only if a difference in the other direction is impossible or irrelevant, and it must be declared before data are seen. Switching to one-sided to rescue p = 0.07 is a form of cheating.
Summary
Frame H0 (no difference) and H1; choose α before analysing.
The p-value is the chance of data this extreme if H0 were true, not the chance H0 is true.
Type I error = false alarm (α); type II error = missed effect (β); power = 1 − β.
A non-significant result with a wide CI is inconclusive, not proof of no effect.
Statistical significance is not clinical importance.
Check your understanding
Draft study guide. Spotted an error or something unclear? Tell us at hello@scintria.com. This is educational material, not clinical or regulatory advice.