Learn / Biostatistics / Logistic and Poisson regression
Prediction models: discrimination, calibration and ROC curves
14 minute read · 6-question quiz
What you will learn
Interpret a ROC curve and the area under it (AUC)
Choose a threshold by balancing sensitivity and specificity
Assess calibration and understand why models must be validated on new data
Logistic regression can also predict: given a patient's characteristics, what is the chance of the outcome? Clinical prediction models (risk scores for cardiovascular disease, sepsis, TB) are built this way. Two questions decide whether one is any use.
- Discrimination: does it rank people correctly (higher predicted risk to those who have the outcome)?
- Calibration: when it says 20%, do about 20% have the outcome?
The ROC curve
Every threshold on the predicted risk makes a different classification: 'high risk if predicted risk ≥ 0.20, 0.30…'. Each gives a sensitivity and a specificity. The receiver operating characteristic (ROC) curve plots sensitivity (true-positive rate) against 1 − specificity (false-positive rate) for every threshold.
The area under the curve (AUC), also called the c-statistic, is the probability that a randomly chosen person with the outcome has a higher predicted risk than a randomly chosen person without it.
| AUC | Interpretation |
| 0.5 | No better than a coin toss |
| 0.6 to 0.7 | Poor |
| 0.7 to 0.8 | Acceptable |
| 0.8 to 0.9 | Good |
| > 0.9 | Excellent (check for overfitting!) |
Our malaria model has AUC = 0.63. All three predictors are statistically significant, yet the model is a weak predictor. That is a fundamental lesson: significant coefficients do not mean accurate prediction.
Calibration
A well-calibrated model's predicted risks match observed frequencies. Plot observed proportion against predicted probability in groups (deciles of risk): points should lie along the diagonal. Calibration is checked with the calibration slope (ideal 1.0) and calibration-in-the-large (ideal 0). The Hosmer-Lemeshow test is often used but depends heavily on sample size; a graph is better.
A model can discriminate well but be badly calibrated (it ranks well but systematically overestimates risk), which matters when the numbers themselves guide decisions.
Validation
A model always fits its own development data better than new data. This 'optimism' grows with the number of predictors tried. To assess true performance:
- Internal validation: bootstrap or cross-validation within the development data.
- External validation: test on a different population, place or time. Many published models lose accuracy sharply when validated elsewhere, and a risk score developed in a high-income hospital may be miscalibrated in rural Uganda.
- Follow the TRIPOD reporting guideline for prediction model studies.
Summary
ROC curves plot sensitivity against 1 − specificity over all thresholds; AUC summarises discrimination.
Significant predictors do not guarantee good prediction; AUC 0.63 is weak.
The best threshold depends on the consequences of false positives and false negatives.
Calibration (predicted vs observed) is as important as discrimination.
Always validate, ideally externally; follow TRIPOD.
Check your understanding
Draft study guide. Spotted an error or something unclear? Tell us at hello@scintria.com. This is educational material, not clinical or regulatory advice.