Research Reproducibility: How Analyst Choices Shape Scientific Truth
A massive crowdsourced reanalysis of 100 social science studies reveals that single-path data analysis often fails to produce consistent results. When multiple independent analysts examine the exact same data, they reach the same statistical conclusion only 74% of the time.
By Aczel, Szaszi, Clelland, Kovacs, Holzmeister, van Ravenzwaaij, Schulz-Kümpel, Hoffmann, Nilsonne, Kosa, Torma, Abdelfatah, Aberson, Acar, Acem, Adamkovic, Adamovich, Adiasto, Ahnström, Akil, Al-Busaidi, Al-Hoorie, Albers, Allen, Alsalti, Altman, Alzahawi, Ambrosini, Anafinova, Anand, Angerer, Angulo-Brunet, Antonietti, Arato, Arenas, Aviña, Azevedo, Bachl, Bago, Bahník, Baker, Balayan, Baldwin, Banai, Banas, Bartoš, Baskin, Bastiaansen, Bault, Bauman, Beazer, Behnke, Bendixen, Berger, Bernard, Bernardic, Bloom, Boldt, Bosch-Rosa, Botvinik-Nezer, Bouyamourn, Bozkurt, Brehm, Breuer, Briggs, Brohmer, Buchanan, Buckenmaier, Buckley, Buczny, Burghart, Butt, Byrd, Cafarelli, Callahan, Capitán, Carriere, Cataldo, Cepaluni, Chan, Chandler, Chang, Chen, Chen, Chen, Chen, Chirkov, Cialfi, Clarke, Coelho, Cohen, Collins, Cook, Corlazzoli, Cummins, Czymara, D’hondt, Rosa, Davis, Davis, Day, De Keyzer, de Leeuw, de Vries, Debnath, Dechterenko, Demiral, Desgroseilliers, Dianovics, Diveica, Dochow-Sondershaus, Dohle, Dong, Dora, Dorrough, Dreber, Du, Edlund, Eerland, Efendić, Elder, Elsherif, Ernst, Estrada, Eudave, Evans, Farrera, Ferrouhi, Fiala, Fialho, Fiechter, Fišar, Flores-Kanter, Folwarczny, Fossum, Franco, Freichel, Freire, Frese, Furnas, Gaebler, Gajary, Galang, Ganschow, Garrison, Gasiorowska, Ponne, Gauriot, Geminiani, Geraldes, Gernsbacher, Giani, Glerean, Gligorić, Gnambs, Godefroidt, González-Bustamante, Goreis, Graf-Vlachy, Grieder, Grigoryev, Grinschgl, Grüning, Guassi Moreira, Guichet, Gurgand, Habibnia, Hafenbrack, Hafenbrädl, Häffner, Hagemeister, Haigh, Hajdu, Hajimoladarvish, Hall, Hamjediers, Hardwick, Harma, Harp, Hartvig, Heiberger, Heim, Hernæs, Hernaus, Heyman, Hicks, Hogeveen, Höpler, Houlihan, Huber, Hughes, Hummler, Huth, Ingendahl, Ishii, Isler, Izydorczak, Jackson, Jahn, Jain, Jakubow, Jang, Jang, Jekel, Jia, Jiménez-Leal, Johnson, Jones, Jungkunz, Kačmár, Kaiser, Kalaycı, Kantorowicz, Karabulut, Karch, Karimi-Rouzbahani, Karl, Kažemekaitytė, Kazlou, Kekecs, Kim, Kirchler, Kiss-Dobronyi, Klasmeier, Klein, Koba, Kołczyńska, Kolias, Kolouch Grabovský, Korbmacher, Korda, Kowal, Kretzschmar, Krivoshchekov, Krypotos, Kubsch, Kunisato, Lacko, Landwehr, Lange, Lee, Lee, Lee, Lemay, Lempert, Leo, Lesage, Levin, Li, Lin, Lindsay, Lisovoj, Liu, Liu, Liu, Iacono, Lodder, López-Bueno, Lopez-Nicolas, Loter, Lou, Lovakov, Lu, Ludwig, Luebber, Lukavský, Luo, Lyu, Maassen, Máčel, Mack, Madan, Mädebach, Maffly-Kipp, Mallinson, Marchetti, Marghetis, Marini, Fages, Martínez, Martinoli, Masiliunas, Massoni, Mathieu, Mayer, Mayer, Mayer, McCormick, McDonough, McGowan, McIntyre, McKee, Meier, Meier, Melero, Merkle, Merz, Michaelides, Michaelsen, Mikolajczak, Mill, Millroth, Miroshnik, Misiak, Mora, Moreau, Moreh, Morvinski, Mushtaq, Nagy, Nater, Naumann, Navarrete, Nebe, Nedderhoff, Nennstiel, Neugebauer, Nicolaisen-Sobesky, Nielsen, Niso, Nowak, Okan, Ong, Onicas, Oswald, Otten, Pandey, Pantazi, Papale, Pärnamets, Pauer, Pavlov, Pawel, Peelle, Peetz, Peez, Pesciarelli, Peterson, Petruželka, Petter, Pfänder, Pfuhl, Phillips, Pietryka, Pirrone, Pit, Plachti, Plank, Ploner, Poldrack, Pollmann, Porcher, Präg, Pua, Pugel, Puri, Püski, Radkani, Raes, Rafaï, Raiber, Rathje, Rehms, Reshetnikov, Reynolds, Reynolds, Rigaud, Rioux, Rivera, Robertson, Román-Caballero, Ropovik, Röseler, Ross, Rotella, Rüffer, Rusche, Rusconi, Russo, Sahm, Salamon, Samahita, Sanaei, Sangchooli, Sarafoglou, Scandola, Schaak, Schaerer, Schares, Schilling, Schmalz, Schmidt, Schonberg, Schreiner, Schröder, Schubert, Schuetze, Schultz, Schulze, Schwartz, Schwitter, Scoggins, Seetahul, Seri, Shanks, Shaw, Shaw, Shen, Siemroth, Sladekova, Somo, Sondhi, Sonmez, Spantig, Speekenbrink, Stamos, Stasielowicz, Steckermeier, Steinkamp, Stoevenbelt, Street, Suchow, Sunde, Sundquist, Suschevskiy, Swain, Szecsi, Szekely-Copîndean, Szumowska, Tacconelli, Talbert, Tang, Tendeiro, Testori, Toffalini, Tomašević, Topel, Torkkeli, Tozzi, Traczyk, Trinidad, Trübutschek, Turek, Uhlich, Uhlmann, Urbanska, Van Assche, van Assen, van Dongen, van Lieshout, van Veldhuizen, Varga, Vaughn, Venczel, Vezzoli, Vierus, Visalli, Voldal, Votta, Wagenmakers, Waldendorf, Walker, Wall, Wallen, Wang, Wang, Wang, Weinmann, Weiß, Westheide, Wichman, Wilcke, Williams, Wisniewski, Woiczyk, Woźniak, Wright, Youyou, Wulff, Yang, Yeung, Yuen, Zawistowski, Zein, Zhao, Zheng, Zhou, Ziller, Zimmerman, Zogmaister, Zultan, Fox, Errington, Nosek
