Statistics in Medicine

Quantifying bias in a health survey: Modeling total survey error in the National Immunization Survey

Journal Article

  • Author(s): NoelleAngelique M. Molinari, Kirk M. Wolter, Benjamin Skalland, Robert Montgomery, Meena Khare, Philip J. Smith, Martin L. Barron, Kennon Copeland, Kathleen Santos, James A. Singleton
  • Article first published online: 04 Feb 2011
  • DOI: 10.1002/sim.3911
  • Read on Online Library
  • Subscribe to Journal


Random‐digit‐dial telephone surveys are experiencing both declining response rates and increasing under‐coverage due to the prevalence of households that substitute a wireless telephone for their residential landline telephone. These changes increase the potential for bias in survey estimates and heighten the need for survey researchers to evaluate the sources and magnitudes of potential bias.

We apply a Monte Carlo simulation‐based approach to assess bias in the NIS, a land‐line telephone survey of 19–35 month‐old children used to obtain national vaccination coverage estimates. We develop a model describing the survey stages at which component nonsampling error may be introduced due to nonresponse and under‐coverage. We use that model and components of error estimated in special studies to quantify the extent to which noncoverage and nonresponse may bias the vaccination coverage estimates obtained from the NIS and present a distribution of the total survey error.

Results indicated that the total error followed a normal distribution with mean of 1.72 per cent(95 per cent CI: 1.71, 1.74 per cent) and final adjusted survey weights corrected for this error. Although small, the largest contributor to error in terms of magnitude was nonresponse of immunization providers. The total error was most sensitive to declines in coverage due to cell phone only households. These results indicate that, while response rates and coverage may be declining, total survey error is quite small. Since response rates have historically been used to proxy for total survey error, the finding that these rates do not accurately reflect bias is important for evaluation of survey data. Published in 2011 by John Wiley & Sons, Ltd.

Related Topics

Related Publications

Related Content

Site Footer


This website is provided by John Wiley & Sons Limited, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ (Company No: 00641132, VAT No: 376766987)

Published features on are checked for statistical accuracy by a panel from the European Network for Business and Industrial Statistics (ENBIS)   to whom Wiley and express their gratitude. This panel are: Ron Kenett, David Steinberg, Shirley Coleman, Irena Ograjenšek, Fabrizio Ruggeri, Rainer Göb, Philippe Castagliola, Xavier Tort-Martorell, Bart De Ketelaere, Antonio Pievatolo, Martina Vandebroek, Lance Mitchell, Gilbert Saporta, Helmut Waldl and Stelios Psarakis.