Increasing interrater reliability using composite performance measures

Wilmot, M. P., Wiernik, B. M., Kostal, J. W.
Industrial and Organizational Psychology, 7(4), 539–542.

LeBreton, Scherer, and James (2014) raise important questions about the implications of unreliable criterion measurement in industrial–organizational (I–O) psychology. Although we do not share the authors’ bleak outlook concerning the state of cumulative knowledge in our field, we do agree that continuing to rely on a criterion with an average reliability of .52 (i.e., single-supervisor rating of overall job performance) is undesirable for the continued advancement of the science and practice of I–O psychology. However, contrary to the conclusions of LeBreton and colleagues, improving criterion reliability need not require abandoning supervisory ratings. Rather, by applying a contemporary understanding of the structure of job performance (Campbell, 2012), reliable and construct-valid supervisory ratings can indeed be obtained. In this commentary, we estimate the interrater reliability of a multidimensional job performance measure and demonstrate that the composite ratings derived from such a measure may ameliorate many of the concerns that LeBreton et al. raise.

[Order of authorship is arbitrary.]