Insights into the accuracy of social scientists’ forecasts of societal change

Grossmann I, Rotella A, Hutcherson CA, Sharpinskyi K, Varnum ME, Achter S, Dhami MK, Guo XE, Kara-Yakoubian M, Mandel DR, Raes L, Tay L, Vie A, Wagner L, Adamkovic M, Arami A, Arriaga P, Bandara K, Baník G, Bartoš F, Baskin E, Bergmeir C, Białek M, Børsting CK, Browne DT, Caruso EM, Chen R, Chie BT, Chopik WJ, Collins RN, Cong CW, Conway LG, Davis M, Day MV, Dhaliwal NA, Durham JD, Dziekan M, Elbaek CT, Shuman E, Fabrykant M, Firat M, Fong GT, Frimer JA, Gallegos JM, Goldberg SB, Gollwitzer A, Goyal J, Graf-Vlachy L, Gronlund SD, Hafenbrädl S, Hartanto A, Hirshberg MJ, Hornsey MJ, Howe PD, Izadi A, Jaeger B, Kačmár P, Kim YJ, Krenzler R, Lannin DG, Lin HW, Lou NM, Lua VY, Lukaszewski AW, Ly AL, Madan CR, Maier M, Majeed NM, March DS, Marsh AA, Misiak M, Myrseth KOR, Napan JM, Nicholas J, Nikolopoulos K, O J, Otterbring T, Paruzel-Czachura M, Pauer S, Protzko J, Raffaelli Q, Ropovik I, Ross RM, Roth Y, Røysamb E, Schnabel L, Schütz A, Seifert M, Sevincer AT, Sherman GT, Simonsson O, Sung MC, Tai CC, Talhelm T, Teachman BA, Tetlock PE, Thomakos D, Tse DC, Twardus OJ, Tybur JM, Ungar L, Vandermeulen D, Vaughan Williams L, Vosgerichian HA, Wang Q, Wang K, Whiting ME, Wollbrant CE, Yang T, Yogeeswaran K, Yoon S, Alves VR, Andrews-Hanna JR, Bloom PA, Boyles A, Charis L, Choi M, Darling-Hammond S, Ferguson ZE, Kaiser CR, Karg ST, Ortega AL, Mahoney L, Marsh MS, Martinie MF, Michaels EK, Millroth P, Naqvi JB, Ng W, Rutledge RB, Slattery P, Smiley AH, Strijbis O, Sznycer D, Tsukayama E, van Loon A, Voelkel JG, Wienk MN, Wilkening T (2023)

Publication Type: Journal article

Publication year: 2023

Journal

Book Volume: 7

Pages Range: 484-501

Journal Issue: 4

DOI: 10.1038/s41562-022-01517-1

Abstract

How well can social scientists predict societal change, and what processes underlie their predictions? To answer these questions, we ran two forecasting tournaments testing the accuracy of predictions of societal change in domains commonly studied in the social sciences: ideological preferences, political polarization, life satisfaction, sentiment on social media, and gender–career and racial bias. After we provided them with historical trend data on the relevant domain, social scientists submitted pre-registered monthly forecasts for a year (Tournament 1; N = 86 teams and 359 forecasts), with an opportunity to update forecasts on the basis of new data six months later (Tournament 2; N = 120 teams and 546 forecasts). Benchmarking forecasting accuracy revealed that social scientists’ forecasts were on average no more accurate than those of simple statistical models (historical means, random walks or linear regressions) or the aggregate forecasts of a sample from the general public (N = 802). However, scientists were more accurate if they had scientific expertise in a prediction domain, were interdisciplinary, used simpler models and based predictions on prior data.

Involved external institutions

Pavol Jozef Šafárik University in Kosice / Univerzita Pavla Jozefa Šafárika v Košiciach Slovakia (SK) University of Cambridge United Kingdom (GB) University of Dayton United States (USA) (US) Hermes Germany GmbH Germany (DE) Universidade Lusófona Portugal (PT) Illinois State University (ISU) United States (USA) (US) National Pingtung University Taiwan (TW) University of Arizona United States (USA) (US) University of Victoria (UVic) Canada (CA) Singapore Management University (SMU) Singapore (SG) Columbia University United States (USA) (US) California State University (CSUF) United States (USA) (US) Loma Linda University (LLU) United States (USA) (US) University of Nottingham United Kingdom (GB) Carnegie Mellon University United States (USA) (US) Singapore University of Social Sciences SUSS / 新加坡社科大學 Singapore (SG) Yale University United States (USA) (US) Monash University Australia (AU) University of Washington United States (USA) (US) University of Zurich / Universität Zürich (UZH) Switzerland (CH) Oklahoma State University United States (USA) (US) University of Hawaiʻi-West Oʻahu United States (USA) (US) Stanford University United States (USA) (US) University of Waterloo (UW) Canada (CA) Technische Universität Dortmund Germany (DE) University of Oklahoma United States (USA) (US) Cornell University United States (USA) (US) Harvard University United States (USA) (US) University of Wisconsin - Madison United States (USA) (US) University of Queensland Australia (AU) University of St Andrews United Kingdom (GB) UMass University of Massachusetts Dartmouth United States (USA) (US) University of North Carolina Wilmington (UNCW) United States (USA) (US) Vrije Universiteit Amsterdam (VU) / University Amsterdam Netherlands (NL) University of Canterbury New Zealand (NZ) Boston University United States (USA) (US) University College London (UCL) United Kingdom (GB) Pusan National University (PNU) Korea, Republic of (KR) Florida State University (FSU) United States (USA) (US) Georgetown University United States (USA) (US) University of California Los Angeles (UCLA) United States (USA) (US) Durham University United Kingdom (GB) Aberystwyth University United Kingdom (GB) Aarhus University Denmark (DK) University of Agder / Universitetet i Agder / Agderin yliopisto Norway (NO) University of Silesia in Katowice / Uniwersytet Śląski w Katowicach Poland (PL) University of Amsterdam Netherlands (NL) Central Connecticut State University CCSU United States (USA) (US) Wright State University United States (USA) (US) Univerzita Karlova v Praze / Charles University in Prague Czech Republic (CZ) Macquarie University Australia (AU) Georgia Southern University United States (USA) (US) University of Haifa / אוניברסיטת חיפה Israel (IL) University of Oslo Norway (NO) BI Norwegian Business School / Handelshøyskolen BI Norway (NO) The University of Melbourne Australia (AU) Otto-Friedrich-Universität Bamberg Germany (DE) University of California, Berkeley (UC Berkeley) United States (USA) (US) Universität Hamburg (UHH) Germany (DE) Karolinska Institute Sweden (SE) University of Southampton United Kingdom (GB) University of Chicago United States (USA) (US) University of Virginia (UVA) United States (USA) (US) University of Pennsylvania (UPenn) United States (USA) (US) National and Kapodistrian University of Athens Greece (GR) Uppsala University Sweden (SE) University of Strathclyde United Kingdom (GB) Nottingham Trent University United Kingdom (GB) Northwestern University United States (USA) (US) University of Groningen / Rijksuniversiteit Groningen Netherlands (NL) National Research University - Higher School of Economics (HSE) / Высшая школа экономики Russian Federation (RU) Radboud University Nijmegen Netherlands (NL) Ontario Institute for Cancer Research Canada (CA) The University of Winnipeg / Université de winnipeg Canada (CA) Iscte – Instituto Universitário de Lisboa (ISCTE-IUL) / Instituto Superior de Ciências do Trabalho e da Empresa Portugal (PT) Grove City College United States (USA) (US) University of Guelph (U of G) Canada (CA) Adam Mickiewicz University / Uniwersytet im. Adama Mickiewicza w Poznaniu (UAM) Poland (PL) University of Wroclaw / Uniwersytet Wrocławski Poland (PL) University of York United Kingdom (GB) IE University Spain (ES) Hebrew University of Jerusalem Israel (IL) Siena College United States (USA) (US) Memorial University of Newfoundland (MUN) Canada (CA) University of British Columbia Canada (CA) Dominican University of California United States (USA) (US) Tamkang University / 淡江大學 Taiwan (TW) Purdue University United States (USA) (US) University of Oxford United Kingdom (GB) University of Prešov / Prešovská univerzita v Prešove Slovakia (SK) Michigan State University United States (USA) (US) Defence Research and Development Canada Canada (CA) Northumbria University United Kingdom (GB) University of Toronto Canada (CA) Arizona State University (ASU) United States (USA) (US) Saint Joseph's University United States (USA) (US) Technische Universität Hamburg-Harburg (TUHH) Germany (DE) Middlesex University United Kingdom (GB) University of California, San Diego (UC San Diego, UCSD) United States (USA) (US) Toronto Metropolitan University / Ryerson University Canada (CA) Tilburg University Netherlands (NL)

How to cite

APA:

Grossmann, I., Rotella, A., Hutcherson, C.A., Sharpinskyi, K., Varnum, M.E., Achter, S.,... Wilkening, T. (2023). Insights into the accuracy of social scientists’ forecasts of societal change. Nature Human Behaviour, 7(4), 484-501. https://doi.org/10.1038/s41562-022-01517-1

MLA:

Grossmann, Igor, et al. "Insights into the accuracy of social scientists’ forecasts of societal change." Nature Human Behaviour 7.4 (2023): 484-501.

BibTeX: Download