Data analysis is sometimes seen as the mechanical, unimaginative process of revealing results from a research study. it is easy to overlook the fact that results may depend on the chosen analytic strategy, which itself is imbued with theory, assumptions, and choice points. This paper presents the findings of 61 analysts observing 29 football teams, using the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players. The analytic techniques chosen ranged from simple linear regression to complex multilevel reg and Bayesian approaches.
Compared with many research questions, this one is of relatively modest complexity. And yet the process of translating this question from natural language to statistical models gave rise to many different assumptions and choices that influenced the conclusions. This raises the possibility that hidden uncertainty due to the wide range of analytic choices available to researchers exists across a wide variety of research applications. Crowdsourcing data analysis, a strategy in which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective, analytic choices influence research results.
Twenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players. Analytic approaches varied widely across the teams, and the estimated effect sizes ranged from 0.89 to 2.93 (Mdn = 1.31) in odds-ratio units. Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not observe a significant relationship. Overall, the 29 different analyses used 21 unique combinations of covariates. Neither analysts’ prior beliefs about the effect of interest nor their level of expertise readily explained the variation in the outcomes of the analyses. Peer ratings of the quality of the analyses also did not account for the variability. These findings suggest that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy in which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective, analytic choices influence research results.
Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affects Results
R. Silberzahn, E. L. Uhlmann, D. P. Martin, P. Anselmi, F. Aust5, E. Awtrey6, Š. Bahník, F. Bai, C. Bannard9, E. Bonnier10, R. Carlsson11, F. Cheung. G. Christensen, R. Clay14, M. A. Craig1, A. Dalla Rosa, L. Dam, M. H. Evans, I. Flores Cervantes18, N. Fong19, M. Gamez-Djokic20, A. Glenz S. Gordon-McKeon22, T. J. Heaton23, K. Hederos24, M. Heene A. J. Hofelich Mohr, F. Högden, K. Hui27, M. Johannesson10, J. Kalodimos28, E. Kaszubowski29, D. M. Kennedy30, R. Lei15, T. A. Lindsay26, S. Liverani31, C. R. Madan32, D. Molden33, E. Molleman16, R. D. Morey34, L. B. Mulder16, B. R. Nijstad16, N. G. Pope, B. Pope36, J. M. Prenoveau, F. Rink16, E. Robusto4, H. Roderique, A. Sandberg, E. Schlüter, F. D. Schönbrodt25, M. F. Sherman37, S. A. Sommer, K. Sotak, S. Spain C. Spörlein, T. Stafford, L. Stefanutti4, S. Tauber, J. Ullrich21, M. Vianello, E.-J. Wagenmakers, M. Witkowiak, S. Yoon, B. A. Nosek3,
First Published August 23, 2018 Research Article
Advances in Methods and Practices in Psychological Science