Clarifying terminology that describes reproducibility


  • Author: Statistics Views
  • Date: 18 August 2015
  • Copyright: Image appears courtesy of Getty Images

The 2015 August issue of Nature Methods includes a publication titled: 'Clarifying the terminology that describes scientific reproducibility', by Ron Kenett and Galit Shmueli. Two main points come out from reading it:

1. Terminology such as reproducibility, repeatability and replicability has different meanings in different contexts, with confusing consequences.

2. A careful evaluation of a specific study, disclosing how its findings can be generalized, provides clarification into the intent of the terms used.

thumbnail image: Clarifying terminology that describes reproducibility

Specifically, industry is accustomed to use Gage R&R to describe the assessment of the measurement error generated by various testers and measurement systems. In animal behaviour studies, reproducibility is about repeating the same experiments in an attempt to recreate the results. In general, reproducibility, replicability or repeatability are defined by which experimental conditions are changed vs. which are kept constant. Statistical generalizability refers to inferring from a sample to a target population.

Statistical analyses performed in scientific studies are typically aimed at achieving statistical generalizability. Scientific generalizability, on the other hand, refers to applying a model based on a particular context to other situations. Kenett and Shmueli show that reproducibility, repeatability and replicability are aimed at assuring generalizability of one type or another and propose that presenting the generalizability of a study provides the necessary clarification of the terms used.

Generalization is one of the eight dimensions of the Information Quality (InfoQ) framework of applied research they propose in a JRSS(A) paper of 2014. These dimensions consist of 1) Data Resolution, 2) Data Structure, 3) Data Integration, 4) Temporal Relevance, 5) Generalizability, 6) Chronology of Data and Goal, 7) Operationalization, and 8) Communication. A comprehensive treatment of this topic, with examples, will be provided in their forthcoming book titled Information Quality (InfoQ): The Potential of Data and Analytics to Generate Knowledge, that is due out in 2016 and shall be published by Wiley.


Kenett RS & Shmueli G (2015), Clarifying the terminology that describes scientific reproducibility, Nature Methods, 12(8), p. 699.

Kenett, R.S. and Shmueli, G. (2014) On Information Quality, Journal of the Royal Statistical Society, Series A (with discussion), Vol. 177, No. 1, pp. 3-38

Related article: Statistics Views feature article on reproducible research

Related Topics

Related Publications

Related Content

Site Footer


This website is provided by John Wiley & Sons Limited, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ (Company No: 00641132, VAT No: 376766987)

Published features on are checked for statistical accuracy by a panel from the European Network for Business and Industrial Statistics (ENBIS)   to whom Wiley and express their gratitude. This panel are: Ron Kenett, David Steinberg, Shirley Coleman, Irena Ograjenšek, Fabrizio Ruggeri, Rainer Göb, Philippe Castagliola, Xavier Tort-Martorell, Bart De Ketelaere, Antonio Pievatolo, Martina Vandebroek, Lance Mitchell, Gilbert Saporta, Helmut Waldl and Stelios Psarakis.