3 Humans + 1 Computer = Best Prediction

Computers often beat people at predicting the future, as the political analyst Nate Silver convincingly demonstrated during the 2012 U.S. presidential election and shows in his book The Signal and the Noise. But studies have found that humans’ predictions are sometimes better than machines’. So let’s say you’re trying to forecast the success of a product at launch. Should you rely on a computer or trust an expert’s wisdom? New research suggests that the best approach isn’t either-or; it’s both. But “both” doesn’t mean a simple 50/50 mix. In relatively unambiguous contexts, rely more on computer analysis. In highly uncertain ones, average the opinions of three experts and give greater weight to their combined judgment than to the machine’s results. We discovered these guidelines by trying various combinations of human and computer predictions about hit songs on the pop charts in Germany and the UK. Over the course of 12 weeks we asked 180 people—half of them music-industry professionals, half of them graduate students with no particular knowledge of the music business—to predict the Top 100 positions of singles by established and new artists. Making a prediction about a song by an established artist is a “well-structured” problem—past performance data reduce the uncertainty. Here, we found that in a straight contest of man versus machine, the machine tended to win (by “machine” we mean software that relies on common statistical tools to analyze a presumed linear relationship). But the best results came from mixing human and computer predictions. The level of the humans’ expertise was irrelevant; we got the most-accurate results simply by giving the machine’s forecast slightly more weight than the humans’. Improving Your Forecast In our experiment involving pop songs, the best predictions in a highly uncertain context came from mixing human and computer input in differing amounts, depending on the humans’ knowledge. For unknown artists—a more uncertain context—the humans tended to beat the machine. But again, combining computer and human predictions produced the best results. And in this case expertise mattered a lot. When we looked at just the students, the optimal combination gave far more weight to the computer prediction. For the music-industry pros, it was the opposite. We then averaged the judgments of varying numbers of pros. The bigger the group, the better the forecast, but we got the largest gain when going from two experts to three. Prior studies have been contradictory, with computers usually winning in lab experiments and people winning in natural settings. We believe there’s a reason for that. Lab environments tend to be well structured, which favors computers’ systematic processing. The ill-structured conditions in many natural settings favor the messy workings of the human brain. That’s an important point when you’re deciding how much to rely on a computer prediction about a product launch. If it’s a groundbreaking product—something customers haven’t seen before—a computer can provide valuable insights, but you should lean more heavily on the judgment of experienced people.

3 Humans + 1 Computer = Best Prediction

Partner Center