Skip to content
Scintria
ToolsGrant writerFunding
HomeLiteratureClinical trialsEvidence LabProtocol packData analysisLearnProjectsLibraryReferencesApplications
Start free
ScintriaStart free

Learn / Biostatistics / Hypothesis testing

The logic of a significance test

14 minute read · 6-question quiz

Download this lesson as a colour Word or PDF study guide with the figures and answers. Included in paid plans; reading and quizzes stay free.Sign in

What you will learn

  • State null and alternative hypotheses for a research question

  • Define the p-value correctly and say what it does not mean

  • Distinguish type I and type II errors and explain significance level, power and their trade-off

A trial compares systolic blood pressure after 12 weeks on a new regimen with usual care. The groups differ by 6.7 mmHg. Is that a real effect, or could chance alone, in a sample of this size, produce a gap that large? A hypothesis test answers that question in a structured way.

Step by step

  1. 1

    1. State the hypotheses

    The null hypothesis (H0) says there is no difference or association: the true mean difference is 0. The alternative hypothesis (H1) says there is. Most tests are two-sided: H1 is simply 'different', in either direction.

  2. 2

    2. Choose the significance level

    Before looking at the data, decide how much risk of a false alarm you will tolerate. Convention is α = 0.05.

  3. 3

    3. Calculate a test statistic

    Measure how far the observed result is from what H0 predicts, in standard-error units (for example a t or z value).

  4. 4

    4. Find the p-value

    The probability of getting a result at least as extreme as the one observed, if H0 were true.

  5. 5

    5. Decide and report

    If p < α, reject H0 ('statistically significant'); otherwise 'do not reject' (never 'accept'). Report the estimate and CI, not only p.

2.5%2.5%-3-2-10123observed t = −2.05Test statistic if H0 is true
If H0 were true, test statistics would follow this curve. Values in the shaded tails (|z| > 1.96) happen 5% of the time.

What a p-value is, exactly

A p-value is a probability about the data, assuming H0 is true: P(data this extreme | H_0). It is not the probability that H0 is true. A p of 0.03 does not mean a 3% chance the null is right, nor a 97% chance the treatment works.

Two ways to be wrong

Whatever you decide, there are two mistakes you can make.

Decisions and truthH0 trueH0 falseReject H0α (type I)1 − β (power)Keep H01 − αβ (type II)
The four outcomes of a test. α is fixed in advance (usually 5%); power (1 − β) is built in at the design stage through sample size.
  • Type I error (false positive, α): concluding there is an effect when there is none. Controlled by the significance level.
  • Type II error (false negative, β): missing a real effect. Power = 1 − β is the chance of detecting an effect of a given size. Trials aim for 80% to 90%.
Critical valueH0: no differenceH1: real differenceα: false positive (Type I)β: false negative (Type II)Power = 1 − β
The null distribution (left) and the distribution when a real effect exists (right), in standard-error units. Where the effect distribution lies beyond the cut-off, we detect it: 80% power here.

Lowering α (more stringent) reduces false alarms but raises β unless the sample size grows. The cut-off moves right, and more of the real-effect curve falls on the 'miss' side. The only free lunch is a larger sample, a bigger true effect, or less variable measurements.

One-sided or two-sided?

A two-sided test looks for differences in either direction and is the default in clinical research. A one-sided test is justified only if a difference in the other direction is impossible or irrelevant, and it must be declared before data are seen. Switching to one-sided to rescue p = 0.07 is a form of cheating.

'Not significant' is not 'no effect'

A non-significant result in a small trial usually means the study had too little power. Absence of evidence is not evidence of absence. Always inspect the confidence interval: if it spans both trivial and important effects, the study is inconclusive.

Statistical vs clinical significance

A trial of 40,000 patients finds a blood pressure reduction of 0.4 mmHg (95% CI 0.2 to 0.6, p < 0.0001). Highly significant, clinically trivial. A pilot of 30 patients finds 8 mmHg (95% CI -1 to 17, p = 0.08). Not significant, potentially important. Always ask: how big is the effect, and does it matter?

Summary

  • Frame H0 (no difference) and H1; choose α before analysing.

  • The p-value is the chance of data this extreme if H0 were true, not the chance H0 is true.

  • Type I error = false alarm (α); type II error = missed effect (β); power = 1 − β.

  • A non-significant result with a wide CI is inconclusive, not proof of no effect.

  • Statistical significance is not clinical importance.

Check your understanding

Question 1
1

A p-value of 0.03 means:

Question 2
2

A type II error is:

Question 3
3

Power depends on all of these EXCEPT:

Question 4
4

A trial reports p = 0.20 with 95% CI for the difference of -5 to +25 units. Best interpretation?

Question 5
5

If you lower α from 0.05 to 0.01 without changing anything else, power will:

Question 6
6

Choosing a one-sided test after seeing which direction the data went is:

Draft study guide. Spotted an error or something unclear? Tell us at hello@scintria.com. This is educational material, not clinical or regulatory advice.

Confidence intervals for means and proportionsComparing means: the t-tests