Skip to content
Scintria
ToolsGrant writerFunding
HomeLiteratureClinical trialsEvidence LabProtocol packData analysisLearnProjectsLibraryReferencesApplications
Start free
ScintriaStart free

Learn / Epidemiology / Evaluating interventions

Natural experiments and quasi-experimental evaluations

13 minute read · 6-question quiz

Download this lesson as a colour Word or PDF study guide with the figures and answers. Included in paid plans; reading and quizzes stay free.Sign in

What you will learn

  • Explain when randomisation is not possible and what alternatives exist

  • Calculate a difference-in-differences estimate and state its key assumption

  • Describe an interrupted time series and a regression discontinuity design

Many important public health decisions cannot be randomised: national laws, a sugar tax, a district-wide programme, a pandemic lockdown. Quasi-experimental methods use non-random comparisons designed to approximate the counterfactual.

Difference-in-differences (DiD)

Compare the change over time in the intervention group with the change in a comparison group. The comparison group's change estimates what would have happened to the intervention group without the intervention.

Worked example: a community health worker programme

The percentage of children with fever treated within 24 hours, before and after a programme, in districts with and without it:

BeforeAfterChange
Programme districts30%20%−10
Comparison districts28%26%−2

(Here lower is better: 'percentage not treated within 24 hours'.) The programme districts improved by 10 points, but the comparison districts improved by 2 points anyway (secular trend). The programme's effect is the difference in differences: -10 - (-2) = −8 percentage points.

2022.52527.53001Time (0 = before, 1 = after)Not treated within 24 h (%)Programme districtsComparison districts
Difference-in-differences: the gap between the two lines changes by 8 points after the programme.

The parallel trends assumption

DiD is valid only if, without the programme, both groups would have followed parallel trends. Check this using several pre-programme periods. If the programme districts were already improving faster, DiD overstates the effect. Add controls, match comparison areas and test sensitivity.

Interrupted time series (ITS)

When there is no comparison group, use a long series of measurements before and after the intervention. The analysis (segmented regression) estimates the level change at the intervention and the change in slope, after allowing for the pre-existing trend. Typically 8 to 12 or more observations before and after are recommended, with allowance for seasonality and autocorrelation. A controlled ITS adds a comparison series.

Use: effect of a smoking ban on hospital admissions; a stock-out of antimalarials on monthly cases.

Regression discontinuity (RD)

Where eligibility is set by a cut-off on a continuous score (CD4 below 350 for ART in earlier guidelines, an income threshold for cash transfers), compare people just above and just below the threshold. They are almost identical except for treatment, so it approximates randomisation locally.

Other methods

  • Propensity score matching or weighting: balance measured confounders between treated and untreated.
  • Instrumental variables: use something that affects treatment but not the outcome directly (distance to clinic).
  • Synthetic control: build a weighted combination of untreated areas that matches the treated area before the intervention.
  • Natural experiments: exposure assigned by circumstances as-if randomly (a lottery for places, a change in policy at a border).
Difference-in-differencesInterrupted time seriesRegression discontinuityNeeds a comparison groupAssumes parallel trendsSimple, widely usedNeeds many time pointsNo comparison group neededModels trend and levelchangesNeeds a sharp cut-off ruleCompares people near thethresholdLocal effect only
Quasi-experimental designs and their requirements.

Quasi-experimental is not a quality guarantee

These methods rely on assumptions that cannot be proven. State them, test them and report sensitivity analyses. Where an RCT is feasible and ethical, it is usually better.

Reporting

Use the TREND statement (non-randomised designs), and STROBE for observational designs; name the design and state the assumptions explicitly in the methods section.

Summary

  • Quasi-experimental methods approximate the counterfactual when randomisation is not possible.

  • DiD compares changes over time between groups and assumes parallel trends.

  • ITS uses many time points to estimate level and slope changes.

  • Regression discontinuity compares people just above and below an eligibility cut-off.

  • State and test assumptions; use TREND and STROBE reporting.

Check your understanding

Question 1
1

In the worked example the DiD estimate was:

Question 2
2

The key assumption of difference-in-differences is:

Question 3
3

An interrupted time series needs:

Question 4
4

Comparing people just above and below an eligibility threshold is:

Question 5
5

A national sugar tax cannot be randomised. A reasonable evaluation is:

Question 6
6

Quasi-experimental estimates rely on:

Draft study guide. Spotted an error or something unclear? Tell us at hello@scintria.com. This is educational material, not clinical or regulatory advice.

Cluster, stepped-wedge and pragmatic trialsTransmission dynamics: R0, susceptibles and the epidemic curve