Skip to content
Scintria
ToolsGrant writerFunding
HomeLiteratureClinical trialsEvidence LabProtocol packData analysisLearnProjectsLibraryReferencesApplications
Start free
ScintriaStart free

Learn / Epidemiology / Screening and diagnostic tests

Validity and reliability of tests

14 minute read · 6-question quiz

Download this lesson as a colour Word or PDF study guide with the figures and answers. Included in paid plans; reading and quizzes stay free.Sign in

What you will learn

  • Distinguish validity from reliability and give measures of each

  • Calculate and interpret kappa for agreement between observers

  • Choose a cut-off using ROC thinking and state the trade-off

A measurement can be valid (it measures what it claims to) and reliable (it gives the same result on repetition). A scale that always reads 2 kg too heavy is reliable but not valid. A scale that reads randomly high and low is neither.

Validity: comparison with a gold standard

Measures: sensitivity, specificity, predictive values and likelihood ratios, as in chapter 2 of the biostatistics course. They need a reference ('gold') standard applied to everyone in the study: if only test-positives get the gold standard, the study suffers verification bias (sensitivity is inflated). Other validity concerns: spectrum bias (testing only clear-cut cases and healthy controls inflates accuracy; real clinics see borderline patients) and incorporation bias (the test is part of the gold standard). Report studies of diagnostic accuracy with the STARD guideline.

Reliability: agreement between and within observers

  • Inter-observer reliability: do two readers agree on the same sample?
  • Intra-observer reliability: does one reader agree with themselves on a second reading?
  • Test-retest reliability: same test, same person, repeated.

For categorical results, raw percent agreement overstates reliability because observers agree by chance. Cohen's kappa corrects for this:

\kappa = (p_o - p_e)/(1 - p_e)

where p_o is the observed agreement and p_e the agreement expected by chance from the marginal totals.

Worked example: two radiologists

Two readers classify 100 chest X-rays as abnormal or normal.

B: abnormalB: normalTotal
A: abnormal401050
A: normal54550
Total4555100
  • Observed agreement p_o = (40 + 45)/100 = 0.85.
  • Expected by chance: p_e = (50/100)(45/100) + (50/100)(55/100) = 0.225 + 0.275 = 0.50.
  • \kappa = (0.85 - 0.50)/(1 - 0.50) = 0.70.

Agreement is 85%, but half of that could occur by chance alone; the chance-corrected agreement is 0.70.

Below 0.400.41 to 0.600.61 to 0.80Above 0.80Poor to fairToo muchdisagreementRetrain or clarifycriteriaModerateUsable forresearch with careSubstantialOur example: 0.70Almost perfectVery reliable
A common guide to kappa (Landis and Koch). The labels are conventions, not laws.

For continuous measurements (blood pressure by two nurses, haemoglobin by two meters), agreement is assessed with the intraclass correlation coefficient (ICC) and Bland-Altman limits of agreement, not Pearson correlation. For ordered categories use weighted kappa.

Choosing a cut-off

Most tests give a continuous result (CD4, haemoglobin, a risk score). A cut-off turns it into positive and negative, and moving it trades sensitivity against specificity:

  • Lower the threshold (call more people positive): sensitivity rises, specificity falls, more false positives.
  • Raise it: specificity rises, sensitivity falls, more missed cases.

The ROC curve plots the trade-off across all cut-offs; the area under the curve summarises discrimination. The cut-off to adopt depends on consequences: for a screening test for a serious treatable disease, favour sensitivity; for a confirmatory test before a risky treatment, favour specificity.

Rapid test (AUC about 0.90)1 − Specificity (false-positive rate)Sensitivitychance (AUC 0.5)000.50.511
Each point on the curve is a different cut-off. Moving up and to the right catches more cases at the cost of more false positives.

Test accuracy is not the same as patient benefit

A test can be accurate and still fail to improve outcomes if no effective treatment follows, or if results arrive too late. Evaluate diagnostics at several levels: technical performance, diagnostic accuracy, effect on clinicians' decisions, effect on patient outcomes and cost-effectiveness. Point-of-care tests for TB, syphilis, malaria and HIV have been transformative precisely because they shorten the time from test to treatment.

Quick judgment

A kappa of 0.7 or above is generally acceptable for clinical classification; below 0.4 means that observers disagree too often for the test to be useful without retraining or clearer criteria.

Summary

  • Validity is accuracy against a gold standard; reliability is reproducibility.

  • Kappa corrects observed agreement for chance: (po − pe)/(1 − pe).

  • Verification, spectrum and incorporation bias distort diagnostic accuracy studies.

  • Cut-offs trade sensitivity against specificity; choose by consequences.

  • Test accuracy does not guarantee improved patient outcomes.

Check your understanding

Question 1
1

A scale is consistent but always 3 kg too heavy. It is:

Question 2
2

Observed agreement 90% and chance agreement 50% give kappa:

Question 3
3

Applying the gold standard only to people with a positive test causes:

Question 4
4

Lowering the cut-off for a positive result will:

Question 5
5

To compare two continuous measurement methods, use:

Question 6
6

A test with excellent sensitivity and specificity but no available treatment:

Draft study guide. Spotted an error or something unclear? Tell us at hello@scintria.com. This is educational material, not clinical or regulatory advice.

Principles of screening and its hidden biasesEvaluating screening and diagnostic programmes in practice