Statistics in Medicine

A matrix of kappa‐type coefficients to assess the reliability of nominal scales

Journal Article


To investigate the reliability of nominal scales, Kraemer proposed a measurement model from which kappa coefficients could be derived. More recently she suggested a matrix of coefficients as a comprehensive summary of reliability, contrasting this with use of a single summary kappa statistic. The main diagonal of the matrix consists of binary kappa coefficients for each category which measure the reliability of each category relative to all others, while the off‐diagonal elements are correlation coefficients for pairs of categories. The off‐diagonal elements were suggested as measures of confusion between categories. Schouten also suggested coefficients to assess confusion between pairs of categories, which might be used as alternative off‐diagonal elements in a summary matrix. The two types of off‐diagonal element will be compared. It will be shown that Schouten's coefficients can be expressed in terms of the parameters of Kraemer's measurement model and that they are more easily interpreted as measures of confusion. First, the maximum value for Schouten's coefficient is one. Secondly, for any pair of categories, Schouten's coefficient equals the proportionate reduction in the probability of classifying a subject in one category of the pair having previously classified them in the other. Thirdly, where the coefficient for a pair of categories is less than the summary kappa statistic, it will be shown that combining these two categories will increase the value of the summary kappa statistic. The methods of analysis are applied to data from a study of the reliability of psychiatric diagnosis and used to identify pairs of classifications between which there is substantial confusion. © 1998 John Wiley & Sons, Ltd.

Related Topics

Related Publications

Related Content

Site Footer


This website is provided by John Wiley & Sons Limited, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ (Company No: 00641132, VAT No: 376766987)

Published features on are checked for statistical accuracy by a panel from the European Network for Business and Industrial Statistics (ENBIS)   to whom Wiley and express their gratitude. This panel are: Ron Kenett, David Steinberg, Shirley Coleman, Irena Ograjenšek, Fabrizio Ruggeri, Rainer Göb, Philippe Castagliola, Xavier Tort-Martorell, Bart De Ketelaere, Antonio Pievatolo, Martina Vandebroek, Lance Mitchell, Gilbert Saporta, Helmut Waldl and Stelios Psarakis.