COVID-19: Superforecasting the Reopening
A Look Back at Superforecasters’ Performance and Insights from 2020 and 2021
EXECUTIVE SUMMARY
Good Judgment Inc’s Superforecasters once again delivered accuracy and early insight when forecasting on a set of “return to normalcy” questions early in the COVID-19 pandemic. We analyzed the Superforecasters’ performance on nine questions asking about the reopening of entertainment, major sports events, travel, and education. Several objective metrics show they made exceptionally accurate predictions, even during the unprecedented and uncertain early part of 2020, on eight of those nine questions, with the ninth question scoring in the middle of the spectrum. This paper presents a quantitative analysis of these forecasts and also examines four questions in detail as case studies. Specifically, we contrast public statements and information available at the time with the forecast history and commentary of the Superforecasters at the same point in time. The Superforecasters demonstrated the ability to make early, accurate forecasts and provide valuable descriptive and diagnostic comments, making contributions to the public discourse that were valued by governments, non-profits, and private sector companies alike. The Superforecasters’ performance on these questions was in line with their full track record of precise, well-calibrated probability estimates.
INTRODUCTION
The rapid spread of the novel coronavirus (COVID-19) in the United States from March 2020 onwards resulted in closures and lockdowns affecting many areas of life, including business, sports, education, travel, and leisure. Even before these closures and lockdowns, Good Judgment’s professional Superforecasters had been actively forecasting events related to the COVID-19 pandemic.1 Their “eerily accurate predictions”2 were cited in Time magazine and publicized in multiple media interviews.3
This study examines one aspect of Good Judgment’s forecasting on COVID-194—questions that aimed to predict the reopening of entertainment, major sports events, travel, and education. The topics included:
When New York City public schools would resume in-person instruction;
When Germany’s Bundesliga would next hold a match open to the public;
When Disney World’s Magic Kingdom would reopen to the public;
When either the National Football League (NFL) or Major League Baseball (MLB) would next hold a regular season game;
When UCLA would resume in-person instruction;
What passenger traffic at Los Angeles International Airport (LAX) would be in January 2021 as compared to January 2020;
What passenger traffic at London area airports would be in January 2021 as compared to January 2020;
When the US Transportation Security Administration (TSA) would next screen 1.4 million or more travelers per day for seven consecutive days; and
What would happen with the Tokyo Olympics.
METHODOLOGY
While the reopening questions reflected very specific and well-defined events—the first pitch at an MLB game, the first tourist through the turnstiles at Disney World—they were all proxies for what “normal” life looked like before COVID-19. Each of these questions involved highly complex interactions among multiple actors (politicians, health officials, various stakeholders) in an environment of conflicting national, state, and local politics and regulations. Among the many unknowns at the time was when—or whether—an effective vaccine would be found.
While many institutions and individuals were forecasting COVID-19 case counts, mortality rates, and hospitalizations during the pandemic to varying degrees of success, far fewer were forecasting topics related to a return to normalcy.
In assessing the accuracy of the Superforecasters’ forecasts, we combine two approaches:
We describe and utilize intuitive metrics, such as early identification of the eventual answer option and other measures of overall accuracy, to show which questions the Superforecasters were able to forecast accurately from the start and which questions generated the greatest uncertainty.
We examine four of the nine questions in detail and chronologically match the Superforecasters’ forecasts and commentary with media projections and public reporting. This allows us to contrast the Superforecasters’ forecasts with the predictions of others made at the same time.
Measuring Overall Accuracy: Best practices in forecasting include scoring accuracy when outcomes are known, tracking performance over time, and comparing accuracy across different forecasting questions and different forecasters. This has been the standard approach at Good Judgment Inc since 2015.
All forecasts are scored using a squared error score known as a Brier score, which ranges from 0 (perfect accuracy) to 2 (a complete miss). Generally, a score below 0.1 reflects highly accurate forecasting, a score between 0.1 and 0.2 reflects accurate forecasting with some uncertainty, a score between 0.2 to 0.6 reflects forecasting in the gray zone of uncertainty (e.g., forecasting a “50/50 coin flip”), and a score greater than 0.6 implies the forecasters were confident but in the wrong outcome. Brier scores reflect forecasting performance over the life of the question.
To determine accuracy on the nine “return to normalcy” questions, we scored the Superforecasters’ aggregated daily forecasts using three different metrics:
the Brier score for each question,
the number of days the eventual correct outcome had the highest probability forecast among the other answer options, and
the number of days the eventual correct outcome had a probability forecast over 50% probability. These results are shown in Table 1.
Superforecaster performance was better than the gray zone of uncertainty (i.e., Superforecasters achieved a Brier score below 0.2) on eight out of nine questions and was significantly better (earning a highly accurate Brier score below 0.1) on six of the nine questions. For example, on when New York City public schools would resume in-person instruction and when the NFL or MLB would next hold a regular season game, the Superforecasters selected the correct answer option from the start, demonstrating near-perfect analysis.
The question that presented the Superforecasters with the greatest amount of uncertainty was when the public would next be able to attend a Bundesliga game in Germany. In this case, conflicting reports from credible sources had the Superforecasters oscillating between the correct outcome and other answer options. By analyzing the forecasting history of this question and a subset of the other questions, we can gain insight into the collective thinking and approach of the Superforecasters as well as the inherent difficulty of forecasting such complex topics at the time.
CASE STUDY 1: THE RETURN OF MLB OR NFL
Major League Baseball (MLB) cancelled its spring training games just two weeks before the 2020 season was scheduled to open on 26 March. The regular season of the National Football League (NFL) was scheduled to begin on 10 September, but by late March, more and more coaches and front office executives said it would be “nearly impossible” for the season to start on time.5
The Superforecasters received the forecasting question on when the NFL or MLB regular season would start on 24 April 2020. It resolved three months later, on 23 July 2020, when Major League Baseball officially began regular-season games. Almost 530 distinct forecasts were made in this period, favoring the return of the games in June or July 2020 (the eventual outcome). On aggregate, the Superforecaster crowd achieved a near-perfect Brier score of 0.007. To use an aptly appropriate metaphor, Good Judgment’s Superforecasters knocked this forecast out of the park.
In contrast to the Superforecasters’ steady confidence and convergence on the correct outcome, there was no consensus at the time among sports figures, journalists, and health officials on whether a regular season was even possible in 2020 (see Figure 1). In April, Dr. Anthony Fauci, the top infectious disease expert in the US,implied there could be no MLB or NFL in 2020. As late as June 2020, sports news site Deadspin6 suggested the MLB season was “unlikely” to happen.
In making their forecast, the Superforecasters sifted through myriad conflicting reports and were able to identify the main drivers of the eventual outcome: the financial stakes involved, the resumption of salary negotiations, and the restart of professional sports elsewhere (soccer in Europe, baseball in South Korea, and NASCAR at Darlington Raceway).
“Based on financial stakes, I expect there to be some sort of games using a modified approach within the baseball season,” one of the Superforecasters wrote on 24 April 2020.
CASE STUDY 2: REOPENING MAGIC KINGDOM
The world’s most popular theme park, Disney’s Magic Kingdom in Florida, closed “until further notice”7 on 15 March 2020 due to the COVID-19 pandemic. The forecasting question on when the theme park would reopen launched on 24 April 2020 and resolved with the Magic Kingdom’s reopening on 11 July 2020.
While some media analysts had predicted Disney World would not reopen until 20218 and both Disney’s CEO and CFO underscored a high degree of uncertainty, sending the public on a rollercoaster of conflicting news and conjecture, the Superforecasters provided near-prescient forecasts, achieving a 0.043 Brier score. They identified the following signals in their forecasts: financial incentives to reopen, international and domestic precedents (Shanghai Disney Resort, Disney Springs), and the lenient political environment in Florida toward COVID-related closures.
“The State of Florida will be receptive by June,” a Superforecaster wrote on 25 April 2020. “Disney World is a controlled environment, which gives them an advantage for monitoring and testing. Employees are going to be cautious to work in that environment but may be ready by July with precautions.”
“It is such a huge anchor for Florida’s economy, the pressure certainly is on to open ASAP,” another Superforecaster wrote on 28 April 2020.
CASE STUDY 3: GERMANY’S BUNDESLIGA
When Germany’s premier football league, the Bundesliga, resumed its unfinished season on 15 May 2020 with matches held behind closed doors,9 Good Judgment wanted to know when—or if—Germany would allow the general public to attend a match. The forecasting question opened to the Superforecasters on 15 May 2020 and resolved four months later, on 19 September 2020. The Superforecasters made nearly 500 distinct forecasts and earned a 0.2935 Brier score. As shown in Figure 3, the Superforecasters were less confident in this forecast than in all the others in the “return to normalcy” question set.
On the one hand, many Superforecasters projected a September match based on financial considerations, examples of sports being played in front of a limited number of fans internationally, and nascent vaccine optimism.
“If new cases in Germany continue to decline, there’ll be a lot of pressure from fans to allow at least some of them into the stadium,” one Superforecaster wrote on 19 May 2020. “It all depends on the trajectory of COVID-19 in Germany but could well happen between September and December.”
On the other hand, the Superforecasters noted Germany’s cautious approach to public health, the opposition to early reopening, and the government’s intention to maintain a ban on crowded public events until at least 24 October 2020. Beginning in early July, the Superforecasters placed a high probability on the correct answer option, but this changed in mid-August when German Chancellor Angela Merkel ruled out the possibility that fans would return to stadiums. Likewise, Germany’s federal and state health ministers agreed to extend a stadium spectator ban at Bundesliga matches until at least the end of October.10 Given the authority of both Chancellor Merkel and the Health Ministry, the Superforecasters placed a lot of weight on these sources. Nevertheless, to the surprise of many and in contradiction the chancellor’s and the ministry’s statements, the teams would go on to hold games with limited fans in mid-September.
CASE STUDY 4: TOKYO OLYMPICS
After the 2020 Olympic Games in Tokyo were postponed due to the COVID-19 pandemic, on 31 July 2020, Good Judgment asked the Superforecasters whether the Games would take place as scheduled in 2021. The question had 907 distinct forecasts and resolved almost a year later when the Tokyo Games officially began on 23 July 2021.
Throughout this period, public opinion in Japan remained divided on whether to hold, postpone, or cancel the Games. Furthermore, a range of dissonant headlines, speculations, and even allegations that Japan had privately decided to cancel the Games11 all muddied the waters surrounding the future of the event.
Nevertheless, the Superforecasters continued to forecast that the Games would open as scheduled, achieving a 0.109 Brier score on this question. In addition to the anticipated arrival of the vaccines and official statements by Olympic and Japanese officials, the Superforecasters identified the following key factors that would determine the fate of the Tokyo Games: ample time to create a safety plan, lessons learned from other sporting events, the financial investment already made, and the relatively low number of deaths in Japan in earlier waves of COVID-19.
“We will probably have a vaccine by the time of the Olympics,” one Superforecaster wrote on 31 July 2020.
“The IOC will find a way to hold the Olympics in some form. Almost 75% of IOC income comes from selling broadcast rights,” another pointed out on 2 August 2020.
CONCLUSION
The Superforecasters’ ability to make accurate forecasts and provide prescient commentary on a set of “return to normalcy” questions from early in the COVID-19 pandemic is evident—especially when we examine their comments and rationales against the backdrop of media reporting from the same time. While the relief proved to be transitory in later months—as tracked in later questions—our forecasting on pandemic-related questions from the outset demonstrated the Superforecasters’ ability to offer valuable signal where others failed. It was this ability that led Bryan Walsh, the author of Axios Future, to write: “We depend on experts who often have a vested interest in their predictions. Superforecasting offers an empirically grounded alternative at a moment when the future feels more uncertain than ever.”12
Good Judgment, “Forecasting the Pandemic: Dealing with Death and Other Emotional Subjects,” Insights, 4 March 2021. https://goodjudgment.com/forecasting-the-pandemic/
Good Judgment, “The Cost of Overconfidence,” Insights, 3 May 2021. https://goodjudgment.com/the-cost-of-overconfidence/
Tara Law, “Superforecasters Are Making Eerily Accurate Predictions About COVID-19. Our Leaders Could Learn From Their Approach,” Time, 11 June 2020. https://time.com/5848271/superforecasters-covid-19/
Tim Harford, “Superforecasting the Coronavirus,” More or Less, BBC Radio, 18 April 2020. https://www.bbc.co.uk/sounds/play/p089t3yk
Peter Coy, “‘Superforecasters’ Say a Covid-19 Vaccine Is Still a Ways Off,” Bloomberg Businessweek, 27 May 2020. https://www.bloomberg.com/news/articles/2020-05-27/covid-19-vaccine-is-a-long-way-off-forecasters-say
CBSN, “What ‘Superforecasters’ Predict for the COVID-19 Pandemic,” 22 June 2020. https://www.cbsnews.com/video/what-superforecasters-predict-for-the-covid-19-pandemic/ Good Judgment, “Selections from Our Archive,” n.d. https://goodjudgment.com/about/press-news/
In total, professional Superforecasters at Good Judgment Inc forecasted on over 40 COVID-19-related questions that resolved between March 2020 and August 2021 related to (1) return to normalcy; (2) US and worldwide case, hospitalization, and death counts; (3) US stimulus funding; and (4) vaccines and therapeutics.
New York Post, “Coronavirus Has NFL Teams Concerned 2020 Season Won’t Start on Time,” 25 March 2020. https://nypost.com/2020/03/25/coronavirus-has-nfl-teams-concerned-2020-season-wont-start-on-time/
Deadspin, “Coronavirus Is Telling Sports World to Shut It Down. Update: MLB Closes Down Facilities,” 19 June 2020. https://deadspin.com/coronavirus-is-telling-sports-world-to-shut-it-down-1844098379
CNN Business, “Walt Disney World Closes, Paralyzing the Company’s Tourism Empire,” 13 March 2020. https://edition.cnn.com/2020/03/12/media/disney-world-close-coronavirus/index.html
USA Today, “When Will Disney World and Disneyland Reopen? One Analyst Predicts It May Not Be Until 2021,” 21 April 2020. https://www.usatoday.com/story/travel/experience/america/theme-parks/2020/04/21/coronavirus-stock-analyst-predicts-disney-parks-wont-reopen-in-2020/2998426001/
ZDF, “Mehrheit für vollständige Grenzöffnung in EU,” 15 May 2021. https://www.zdf.de/nachrichten/politik/politbarometer-coronavirus-grenzoeffnung-eu-100.html
DW, “Bundesliga: Angela Merkel Rules out Fans in Stadiums for Now—Reports,” 17 August 2020. https://www.dw.com/en/bundesliga-angela-merkel-rules-out-fans-in-stadiums-for-now-reports/a-54600649
DW, “Germany Rejects Fans’ Early Return to Soccer Stadiums,” 10 August 2020. https://www.dw.com/en/germany-rejects-fans-early-return-to-soccer-stadiums/a-54520164
Times, “Japan Looks for a Way out of Tokyo Olympics Because of Covid,” 21 January 2021. https://www.thetimes.co.uk/article/japan-looks-for-a-way-out-of-tokyo-olympics-because-of-virus-lf868xfnd
Bryan Walsh, “The Superforecasting Guide to Accurately Predicting the Coronavirus,” Axios, 12 August 2020. https://www.axios.com/superforecasting-guide-coronavirus-861f8b7f-6d10-4165-be1f-e9ae758853ca.html