Enlightenment Now: The Case for Reason, Science, Humanism, and Progress

Unfortunately the epistemological standards of common sense—we should credit the people and ideas that make correct predictions, and discount the ones that don’t—are rarely applied to the intelligentsia and commentariat, who dispense opinions free of accountability. Always-wrong prognosticators like Paul Ehrlich continue to be canvassed by the press, and most readers have no idea whether their favorite columnists, gurus, or talking heads are more accurate than a chimpanzee picking bananas. The consequences can be dire: many military and political debacles arose from misplaced confidence in the predictions of experts (such as intelligence reports in 2003 that Saddam Hussein was developing nuclear weapons), and a few percentage points of accuracy in predicting financial markets can spell the difference between gaining and losing a fortune.

A track record of predictions also ought to inform our appraisal of intellectual systems, including political ideologies. Though some ideological differences come from clashing values and may be irreconcilable, many hinge on different means to agreed-upon ends and should be decidable. Which policies will in fact bring about things that almost everyone wants, like lasting peace or economic growth? Which will reduce poverty, or violent crime, or illiteracy? A rational society should seek the answers by consulting the world rather than assuming the omniscience of a bloc of opinionators who have coalesced around a creed.

Unfortunately, the expressive rationality documented by Kahan in his experimental subjects also applies to editorialists and experts. The payoffs that determine their reputations don’t coincide with the accuracy of the predictions, since no one is keeping score. Instead, their reputations hinge on their ability to entertain, titillate, or shock; on their ability to instill confidence or fear (in the hopes that a prophecy might be self-fulfilling or self-defeating); and on their skill in galvanizing a coalition and celebrating its virtue.

Since the 1980s the psychologist Philip Tetlock has studied what distinguishes accurate forecasters from the many oracles who are “often mistaken but never in doubt.”⁴⁴ He recruited hundreds of analysts, columnists, academics, and interested laypeople to compete in forecasting tournaments in which they were presented with possible events and asked to assess their likelihoods. Experts are ingenious at wordsmithing their predictions to protect them from falsification, using weasely modal auxiliaries (could, might), adjectives (fair chance, serious possibility), and temporal modifiers (very soon, in the not-too-distant future). So Tetlock pinned them down by stipulating events with unambiguous outcomes and deadlines (for example, “Will Russia annex additional Ukraine territory in the next three months?” “In the next year, will any country withdraw from the Eurozone?” “How many additional countries will report cases of the Ebola virus in the next eight months?”) and having them write down numerical probabilities.

Tetlock also avoided the common fallacy of praising or ridiculing a single probabilistic prediction after the fact, as when the poll aggregator Nate Silver of FiveThirtyEight came under fire for giving Donald Trump just a 29 percent chance of winning the 2016 election.⁴⁵ Since we cannot replay the election thousands of times and count up the number of times that Trump won, the question of whether the prediction was confirmed or disconfirmed is meaningless. What we can do, and what Tetlock did, is compare the set of each forecaster’s probabilities with the corresponding outcomes. Tetlock used a formula which credits the forecaster not just for accuracy but for accurately going out on a limb (since it’s easier to be accurate by just playing it safe with 50-50 predictions). The formula is mathematically related to how much they would win if they put their money where their mouths were and bet on their predictions according to their own odds.