Journal of Evaluation in Clinical Practice

Using randomization tests to assess treatment effects in multiple‐group interrupted time series analysis

Early View


Rationale, aims, and objectives

Interrupted time series analysis (ITSA) is a popular evaluation methodology in which a single treatment unit's outcome is studied over time and the intervention is expected to “interrupt” the level and/or trend of the outcome, subsequent to its introduction. The internal validity of this analysis is strengthened considerably if the treated unit is contrasted with a comparable control group. However, multiple‐group ITSA typically has small sample sizes, and parametric methods for multiple‐group ITSA require strong assumptions that are unlikely to be met, possibly resulting in misleading P values. In this paper, randomization tests are introduced as a non‐parametric, distribution‐free option for computing exact P values.


The effect of California's Proposition 99 (passed in 1988) for reducing cigarette sales is evaluated by comparing California (CA) to Montana (MT) and Idaho (ID)—the two best matched control states not exposed to any smoking reduction initiatives. Results from randomization tests are contrasted to those of interrupted time series analysis regression (ITSAREG)—a commonly used parametric approach for evaluating treatment effects in ITSA studies.


Both approaches found ID and MT to be comparable to CA on their preintervention time series, and both approaches equally found CA to have statistically lower cigarette sales in the postintervention period (P < 0.01).


In these data, randomization tests computed P values comparable with ITSAREG, bolstering confidence in the intervention effect. Routinely including randomization tests as a complement, or alternative, to parametric methods is therefore beneficial because randomization tests are free of assumptions regarding sample size and distribution and are extremely flexible in the choice of test statistic.

Related Topics

Related Publications

Related Content

Site Footer


This website is provided by John Wiley & Sons Limited, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ (Company No: 00641132, VAT No: 376766987)

Published features on are checked for statistical accuracy by a panel from the European Network for Business and Industrial Statistics (ENBIS)   to whom Wiley and express their gratitude. This panel are: Ron Kenett, David Steinberg, Shirley Coleman, Irena Ograjenšek, Fabrizio Ruggeri, Rainer Göb, Philippe Castagliola, Xavier Tort-Martorell, Bart De Ketelaere, Antonio Pievatolo, Martina Vandebroek, Lance Mitchell, Gilbert Saporta, Helmut Waldl and Stelios Psarakis.