Skip to content
Scintria
ToolsGrant writerFunding
HomeLiteratureClinical trialsEvidence LabProtocol packData analysisLearnProjectsLibraryReferencesApplications
Start free
ScintriaStart free

Learn / Biostatistics / Logistic and Poisson regression

Prediction models: discrimination, calibration and ROC curves

14 minute read · 6-question quiz

Download this lesson as a colour Word or PDF study guide with the figures and answers. Included in paid plans; reading and quizzes stay free.Sign in

What you will learn

  • Interpret a ROC curve and the area under it (AUC)

  • Choose a threshold by balancing sensitivity and specificity

  • Assess calibration and understand why models must be validated on new data

Logistic regression can also predict: given a patient's characteristics, what is the chance of the outcome? Clinical prediction models (risk scores for cardiovascular disease, sepsis, TB) are built this way. Two questions decide whether one is any use.

  1. Discrimination: does it rank people correctly (higher predicted risk to those who have the outcome)?
  2. Calibration: when it says 20%, do about 20% have the outcome?

The ROC curve

Every threshold on the predicted risk makes a different classification: 'high risk if predicted risk ≥ 0.20, 0.30…'. Each gives a sensitivity and a specificity. The receiver operating characteristic (ROC) curve plots sensitivity (true-positive rate) against 1 − specificity (false-positive rate) for every threshold.

Model (AUC 0.63)Better (AUC 0.87)1 − Specificity (false-positive rate)Sensitivitychance (AUC 0.5)000.50.511
A model close to the diagonal (the first curve) hardly beats guessing; one that hugs the upper left corner discriminates well.

The area under the curve (AUC), also called the c-statistic, is the probability that a randomly chosen person with the outcome has a higher predicted risk than a randomly chosen person without it.

AUCInterpretation
0.5No better than a coin toss
0.6 to 0.7Poor
0.7 to 0.8Acceptable
0.8 to 0.9Good
> 0.9Excellent (check for overfitting!)

Our malaria model has AUC = 0.63. All three predictors are statistically significant, yet the model is a weak predictor. That is a fundamental lesson: significant coefficients do not mean accurate prediction.

Choosing a threshold

With the malaria model, a cut-off for 'high risk':

Cut-offSensitivitySpecificity
Predicted risk ≥ 0.2078%38%
≥ 0.3037%79%
≥ 0.405%96%

There is no 'best' cut-off in the abstract. If missing a case is dangerous and follow-up is cheap (screening for TB), choose a low threshold for high sensitivity and accept false positives. If a positive result triggers costly or risky treatment, favour specificity. The right balance depends on the consequences of each error, formalised in decision-curve analysis.

Calibration

A well-calibrated model's predicted risks match observed frequencies. Plot observed proportion against predicted probability in groups (deciles of risk): points should lie along the diagonal. Calibration is checked with the calibration slope (ideal 1.0) and calibration-in-the-large (ideal 0). The Hosmer-Lemeshow test is often used but depends heavily on sample size; a graph is better.

A model can discriminate well but be badly calibrated (it ranks well but systematically overestimates risk), which matters when the numbers themselves guide decisions.

Validation

A model always fits its own development data better than new data. This 'optimism' grows with the number of predictors tried. To assess true performance:

  • Internal validation: bootstrap or cross-validation within the development data.
  • External validation: test on a different population, place or time. Many published models lose accuracy sharply when validated elsewhere, and a risk score developed in a high-income hospital may be miscalibrated in rural Uganda.
  • Follow the TRIPOD reporting guideline for prediction model studies.

Do not choose the cut-off on the same data you test on

Selecting the 'optimal' cut-off from the data and then reporting its sensitivity and specificity on the same data produces over-optimistic figures. Validate in new data.

Questions to ask of any prediction model

What is the outcome and time horizon? Was it validated externally? What are the AUC and calibration? How many events per predictor? Is the performance good enough to change a decision, and does the model add to simple clinical judgment?

Summary

  • ROC curves plot sensitivity against 1 − specificity over all thresholds; AUC summarises discrimination.

  • Significant predictors do not guarantee good prediction; AUC 0.63 is weak.

  • The best threshold depends on the consequences of false positives and false negatives.

  • Calibration (predicted vs observed) is as important as discrimination.

  • Always validate, ideally externally; follow TRIPOD.

Check your understanding

Question 1
1

An AUC of 0.50 indicates:

Question 2
2

The ROC curve plots:

Question 3
3

A TB screening tool is used to decide who gets a confirmatory test that is cheap and safe. Which threshold approach is sensible?

Question 4
4

A model predicts 30% risk in a group where 10% have the outcome. This is a problem of:

Question 5
5

Why is internal performance usually optimistic?

Question 6
6

The reporting guideline for prediction model studies is:

Draft study guide. Spotted an error or something unclear? Tell us at hello@scintria.com. This is educational material, not clinical or regulatory advice.

Logistic regression: modelling a yes/no outcomePoisson regression: counts, rates and person-time