Research Synthesis Methods

A tool to assess the quality of a meta‐analysis

Journal Article

  • Author(s): Julian P.T. Higgins, Peter W. Lane, Betsy Anagnostelis, Judith Anzures‐Cabrera, Nigel F. Baker, Joseph C. Cappelleri, Scott Haughie, Sally Hollis, Steff C. Lewis, Patrick Moneuse, Anne Whitehead
  • Article first published online: 18 Oct 2013
  • DOI: 10.1002/jrsm.1092
  • Read on Online Library
  • Subscribe to Journal


Because meta‐analyses are increasingly prevalent and cited in the medical literature, it is important that tools are available to assess their methodological quality. When performing an empirical study of the quality of published meta‐analyses, we found that existing tools did not place a strong emphasis on statistical and interpretational issues.


We developed a quality‐assessment tool using existing materials and expert judgment as a starting point, followed by multiple iterations of input from our working group, piloting, and discussion. After having used the tool for our empirical study, agreement for four key items in the tool was measured using weighted kappa coefficients.


Our tool contained 43 items divided into four key areas (data sources, analysis of individual studies, meta‐analysis methods, and interpretation), and each area ended with a summary question. We also produced guidance for completing the tool. Agreement between raters was fair to moderate.


The tool should usefully inform subsequent initiatives to develop quality‐assessment tools for meta‐analysis. We advocate use of consensus between independent raters when assessing statistical appropriateness and adequacy of interpretation in meta‐analyses. Copyright © 2013 John Wiley & Sons, Ltd.

Related Topics

Related Publications

Related Content

Site Footer


This website is provided by John Wiley & Sons Limited, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ (Company No: 00641132, VAT No: 376766987)

Published features on are checked for statistical accuracy by a panel from the European Network for Business and Industrial Statistics (ENBIS)   to whom Wiley and express their gratitude. This panel are: Ron Kenett, David Steinberg, Shirley Coleman, Irena Ograjenšek, Fabrizio Ruggeri, Rainer Göb, Philippe Castagliola, Xavier Tort-Martorell, Bart De Ketelaere, Antonio Pievatolo, Martina Vandebroek, Lance Mitchell, Gilbert Saporta, Helmut Waldl and Stelios Psarakis.