Learn / Epidemiology / Screening and diagnostic tests
Validity and reliability of tests
14 minute read · 6-question quiz
What you will learn
Distinguish validity from reliability and give measures of each
Calculate and interpret kappa for agreement between observers
Choose a cut-off using ROC thinking and state the trade-off
A measurement can be valid (it measures what it claims to) and reliable (it gives the same result on repetition). A scale that always reads 2 kg too heavy is reliable but not valid. A scale that reads randomly high and low is neither.
Validity: comparison with a gold standard
Measures: sensitivity, specificity, predictive values and likelihood ratios, as in chapter 2 of the biostatistics course. They need a reference ('gold') standard applied to everyone in the study: if only test-positives get the gold standard, the study suffers verification bias (sensitivity is inflated). Other validity concerns: spectrum bias (testing only clear-cut cases and healthy controls inflates accuracy; real clinics see borderline patients) and incorporation bias (the test is part of the gold standard). Report studies of diagnostic accuracy with the STARD guideline.
Reliability: agreement between and within observers
- Inter-observer reliability: do two readers agree on the same sample?
- Intra-observer reliability: does one reader agree with themselves on a second reading?
- Test-retest reliability: same test, same person, repeated.
For categorical results, raw percent agreement overstates reliability because observers agree by chance. Cohen's kappa corrects for this:
\kappa = (p_o - p_e)/(1 - p_e)
where p_o is the observed agreement and p_e the agreement expected by chance from the marginal totals.
For continuous measurements (blood pressure by two nurses, haemoglobin by two meters), agreement is assessed with the intraclass correlation coefficient (ICC) and Bland-Altman limits of agreement, not Pearson correlation. For ordered categories use weighted kappa.
Choosing a cut-off
Most tests give a continuous result (CD4, haemoglobin, a risk score). A cut-off turns it into positive and negative, and moving it trades sensitivity against specificity:
- Lower the threshold (call more people positive): sensitivity rises, specificity falls, more false positives.
- Raise it: specificity rises, sensitivity falls, more missed cases.
The ROC curve plots the trade-off across all cut-offs; the area under the curve summarises discrimination. The cut-off to adopt depends on consequences: for a screening test for a serious treatable disease, favour sensitivity; for a confirmatory test before a risky treatment, favour specificity.
Summary
Validity is accuracy against a gold standard; reliability is reproducibility.
Kappa corrects observed agreement for chance: (po − pe)/(1 − pe).
Verification, spectrum and incorporation bias distort diagnostic accuracy studies.
Cut-offs trade sensitivity against specificity; choose by consequences.
Test accuracy does not guarantee improved patient outcomes.
Check your understanding
Draft study guide. Spotted an error or something unclear? Tell us at hello@scintria.com. This is educational material, not clinical or regulatory advice.