Why AI Can’t Write This Article (Yet)

Roger Schank, a researcher and former professor, once proposed a novel goal for artificial intelligence: A computer should be able to watch West Side Story and recognize the plot of Romeo and Juliet. Schank and his students believed that stories are central to intelligence, reasoning, and meaning. By Schank’s measure, today’s AI isn’t intelligent at all. The feature article on AI that HBR.org published earlier this week is, ironically, a good example of the kind of work that computers can’t yet do. It was written by two experts who drew on decades of experience to formulate a thesis, assemble evidence, and construct a narrative. And three editors helped to shape the nearly 5,000 words that made it into the final piece. The fact that software can’t yet write an article like that isn’t a knock on AI, or evidence that it won’t be transformative. But that fact offers a window into how, exactly, machine learning technologies work, what they are and aren’t good at right now, and how they’ll develop as writing tools — or even writers — in the future. Not Ready for Long-Form Today’s AI works by formulating tasks as prediction problems and then using statistical techniques and lots of data to make predictions. One simple example of a text-based prediction problem is auto-complete. When I type “How’d” into a text message, my phone uses data and statistical modeling to predict what’s coming next. It offers “it,” “you,” or “the.” “It” is what I had in mind, and once I select that, my phone moves on to predicting the next word. This time it’s so confident that I’m going to select “go” (which is right) that it doesn’t even offer other options but instead moves on to the next word, suggesting “go with” or “go today.” In machine learning, prediction problems like this are called supervised learning. Given a data set containing the right answer — in this case lots of completed text messages — an algorithm learns to recognize patterns, such as that “go” often follows “How’d it.” (Another kind of machine learning, unsupervised learning, works differently, but supervised learning has driven most of the recent progress in the field.) The process of writing a magazine feature can’t easily be distilled into a prediction problem, however — at least not yet. As Sam Bowman, a professor at New York University, told a recent conference on AI and journalism, “The notion of really generating long-form coherent text without a very clear, journalist-specified template is quite far away.” Researchers have shown that machine learning can generate coherent text in specific settings, Bowman notes, but “really building systems that are able to go all the way from an abstract idea or a set of facts to a long-form coherent text is still something that’s quite difficult.” To illustrate that difficulty, Bowman pointed to a screenplay, titled Sunspring, written last year using machine learning. The script was generated by feeding dozens of science fiction screenplays into a neural network — a type of machine learning algorithm — at the character level, meaning that the unit of data the algorithm was learning from was a single character of text. Given the characters that had come before, the algorithm was learning to predict which character would come next. Here’s the result. You don’t want to skip this: That the actors in Sunspring are speaking actual English words is itself impressive (even if they don’t make much sense): Before the neural network “read” those scripts, it not only didn’t know how to write a screenplay but also had no knowledge of the English language. It learned some of the features of a screenplay — for instance, that lines of text should be assigned to characters and that stage directions should be included. Again, it learned all this just by reading a few dozen scripts. What it didn’t pick up from all those screenplays was the art of narrative. Sunspring has no story. Its characters exist only in the sense that they’ve been given sentences to speak. The script shows how far machine learning has to go before it masters storytelling, or becomes “intelligent.” Yet the algorithm’s ability to construct sentences and to recognize basic features of a screenplay suggests that AI could play a role in the future of writing. But that future, at least in the near term, is limited. AI-Generated Summaries One area of writing in which machine learning is already making useful progress is summaries. Finding the most important parts of a text and producing a summary of them is an extremely common writing task: Press teams compile “clips” of the day’s news, reporters summarize previous developments while writing a story, think tanks summarize a new study, book editors summarize a chapter. Some of that work can now be done by machines, and startups and tech companies alike are racing to build tools and products to make it more accessible. Auto-summarization techniques usually fit into one of two categories: extractive or abstractive. Extractive methods try to identify the most important sentences in a document and then create a summary by stitching them together. Modern versions of this technique are quite complicated, but the original idea, which Hans Peter Luhn introduced at IBM in 1958, gives a sense of the approach. Luhn proposed that the words used most frequently in a document (excluding very common words such as “the” and “and”) offer clues to the document’s subject. Sentences that contain those common words are therefore most representative of the document; by extracting those sentences and combining them into a paragraph, something approximating a summary can be created. (Even in describing this original approach, I’m oversimplifying. For more, here is an excellent history of the subfield of auto-summarization, by Kathy McKeown, of Columbia University, and Ani Nenkova, of the University of Pennsylvania.) Abstractive summaries, by contrast, attempt to articulate the information contained in one or more documents in original language written by the algorithm. This approach is more ambitious, and until recently it hasn’t worked very well. As the Sunspring script illustrates, generating new language is difficult. But progress in deep learning, a subfield of machine learning, has led to renewed interest in abstractive summarization, and produced some promising results. To illustrate what machine learning can and can’t do, let’s compare an editor-written summary of our AI feature with two automated summaries, one extractive and one abstractive: Three Summaries: Human, Extractive, and Abstractive Here are summaries of the Big Idea article “The Business of Artificial Intelligence,” by Erik Brynjolfsson and Andrew McAfee Human “General-purpose technologies,” such as the internal combustion engine, have been the fundamental drivers of economic growth for 250 years. Artificial intelligence—particularly machine learning (ML)—is the most important such technology of our era. In the coming decade, practically every industry will transform its keys processes and business models to take advantage of ML. But not all expectations surrounding AI are realistic. In this article the authors describe its real potential, its practical implications, and the barriers to its adoption. They note three pieces of good news for organizations looking to put ML to use today: AI skills are spreading quickly, through online educational resources as well as universities; the necessary algorithms and hardware for modern AI can be bought or rented as needed; and companies may not need all that much data to start making productive use of ML. They also note three risks: Machines may have hidden biases, derived from the data used to train them; neural networks deal with statistical rather than literal truths; and diagnosing and correcting system errors is often difficult, because a solution’s underlying structure can be unimaginably complex. Extractive The most important of these are what economists call general-purpose technologies—a category that includes the steam engine, electricity, and the internal combustion engine. Companies as diverse as Walmart, UPS, and Uber found ways to leverage the technology to create profitable new business models. The most important general-purpose technology of our era is artificial intelligence, particularly machine learning (ML)—that is, the machine’s ability to keep improving its performance without humans having to explain exactly how to accomplish all the tasks it’s given. In the sphere of business, AI is poised have a transformational impact, on the scale of earlier general-purpose technologies. Although it is already in use in thousands of companies around the world, most big opportunities have not yet been tapped. What Can AI Do Today? The term artificial intelligence was coined in 1955 by John McCarthy, a math professor at Dartmouth who organized the seminal conference on the topic the following year. The fallacy that a computer’s narrow understanding implies broader understanding is perhaps the biggest source of confusion, and exaggerated claims, about AI’s progress. —Fast Forward Labs Abstractive companies as diverse as walmart, ups, and uber found ways to manipulate the technology to create profitable new business models. in the sphere of business, ai is poised to have a transformational impact, on the scale of earlier general purpose technologies. in the past few years machine learning has become far more effective and widely available. —Alexander Rush The first summary was written by an HBR editor. It’s grammatically correct, it contains the article’s main point, and it speaks in the third person (“the authors describe”). The second summary is extractive and was produced using a prototype built by Fast Forward Labs, a research firm. Using actual articles and summaries from a website of reading recommendations, the Fast Forward team trained a neural network to score sentences according to the likelihood that they would be included in a summary. The highest-scoring sentences, combined in the order in which they appear in the original article, become the summary. In the case of our article, the model’s highest-scoring sentence is the one beginning “The most important general-purpose technology of our era is artificial intelligence,” which is also arguably the article’s thesis. In that sense, the extractive summarizer did well. But when the top seven sentences are arranged in their original order, the first sentence includes the pronoun “these” with no mention of what it refers to. (Teaching these systems to recognize the noun to which a pronoun refers is difficult, and the Fast Forward prototype did not attempt it.) The third summary, courtesy of Alexander Rush, a professor of engineering at Harvard, is abstractive. Rush trained his system to write three-sentence summaries of CNN articles, and although he emphasizes that it isn’t state-of-the-art, he offered to try it on the first 450 words of our AI feature. “The system is, in theory, abstractive,” he says, “so it can generate anything it wants. In practice, it looks like it is generating mostly sentences it sees in the original article itself.” In other words, it avoids the nonsensical results of Sunspring at the cost of originality. And like the extractive summary, this one captures the article’s key themes but includes a reference to “the technology” without providing necessary context. Are these summaries good enough to replace human-written ones? Perhaps not. But that’s not the right question. A better one is whether AI-written first drafts of summaries might speed up our process. And here the answer is almost certainly yes. AI as Research Assistant Summarization may seem too narrow a task to make much of a difference in the writing process, but combined with related technologies, it creates the opportunity to assist writers in a crucial part of their process: research. And research is “the hardest thing we do as writers,” according to David Hill, the editor in chief of SingularityHUB, a niche technology and science publication. Google, whose search algorithms lean on AI, has already transformed the research process and made writers significantly more productive. But Google isn’t a perfect research assistant. Hill describes Google searching as “shallow” and “frenetic.” “It’s horrifically laborious, all the searching you would do,” says Susannah Locke, an editor at Vox.com. She found herself thinking, “Isn’t there something that can do this for me?” Tim Lee, of Ars Technica, describes his process of “unstructured” reading: finding 10 to 15 papers on a subject, reading them, and taking notes. He dreams of a tool that can take 1,000 pages on a topic and identify the 10 pages to start with. The immediate opportunity isn’t to fully automate the research process but to make it more structured and efficient. “I don’t understand why news sites don’t let you just click a name and assemble a backgrounder,” says Brian Ulicny, a data scientist at Thomson Reuters Labs. (Disclosure: Ulicny’s wife and I are colleagues.) In 2006, while at Lycos, Ulicny authored a paper in which he describes an “information fusion engine.” Type in a name or a topic, as you might in Google, and instead of returning a list of links, the system arranges paragraphs from content found across the web into a “coherent summary report or background briefing,” which Ulicny calls “something like the level of the first draft of a Wikipedia article.” Ulicny isn’t the only one to suggest that topic or news overviews can be automatically generated by software. Computer scientists have been building systems and publishing papers along these lines for more than 15 years. These projects are technically complex, and they vary in important ways. But they face the same challenges and follow a similar process. Hilary Mason, a data scientist and the founder of Fast Forward Labs, outlines the main tasks these systems must perform: First, they have to identify source data, meaning some number of text documents such as news articles. Then they need to identify the most important information within those documents and extract it. Finally, that information has to be presented to the end user. Somewhere along the way, many of these systems take a fourth step: They try to identify some structure for the story. Is it a chronology of independent events? A biography of a person? Part of a larger story? Structure can both help the system decide what information is important and provide an outline of how to present it to the end user. This process resembles the way humans approach at least simple research and writing tasks. John O’Neil edits Bloomberg’s explanatory QuickTake, but before that he worked on topic pages at the New York Times. He describes the process he and his team used to write the text for those topic pages (which have since changed format): First, find four or five key articles the Times had published about the subject. Second, identify the background paragraphs in each story (as opposed to the news). Third, write a summary that combined the information from those background paragraphs. At least when writing topic pages, the key steps for both humans and software are the same. The Future of AI and Writing If these tools have been around for years, even in an imperfect form, why haven’t they had more impact on writing? One reason, as with so much revolutionary technology, is culture. On the one hand, many writers don’t perceive a need for these tools; on the other, computer scientists haven’t always been concerned with how people will actually use their work. In auto-summarization, according to Ani Nenkova, the focus has mostly been on improving accuracy, rather than on thinking about how the technology could be embedded in a tool that people would actually use. Money is another factor — lots of writers and newsrooms don’t have much of it. “Most of the progress [in natural language processing] happened when security analysts and the government were interested in being able to monitor foreign news,” says Nenkova, whose PhD was funded by DARPA. Finance, too, is an area in which machine learning and natural language processing have had an impact, in large part because the money has been there to make it happen. The final reason these tools haven’t made more of a dent in writing is simply that the results haven’t been good enough to consistently serve readers well on their own. In his paper, Ulicny describes an auto-generated backgrounder for the retired hockey player Mario Lemieux. The system recognized key subtopics that should be part of the explainer, such as “games,” “seasons,” and “Pittsburgh Penguins.” It also added “ice” — a topic that’s clearly related in some sense but that no writer would include in a profile of a hockey player. All of this is changing. The technology is getting both better and easier to use, and more and more writers and media companies are recognizing that smart software can help them do their work. It’s clear to me that machine learning does have a near-term role in many types of writing, but for the most part it won’t involve producing full-fledged articles. Rather, it will help journalists produce those articles more effectively. Lots of people are working on tools to make that happen. David Hill has a grant to create an open-source research assistant. Frase, an early-stage startup in Boston, is working on something similar, though its founders plan to target content marketers as initial customers. Google Docs already has such a tool, but its utility is limited. Vox built a Slack bot to show writers older articles that they might want to cite in new stories. IBM Watson built a prototype called Watson Angles that summarized news stories, created timelines, and highlighted significant quotations. The prototype, which was removed from the web last fall, also included some key pieces of metadata, such as sentiment analysis of how Reddit users had responded to the news story in question, ranging from positive to negative. These projects are just the beginning. Imagine a news story on the recent fire in London that mentions the fact that your friend who lives there posted an hour ago that she was safe. Or text that automatically adjusts to the reader’s level of background knowledge. Or fact-checking built into a word processor. Or topic pages covering a long tail of niche subjects that smaller audiences are passionate about but that few publishers today can afford to produce. Or a research assistant that recalls a relevant story written a century ago just as readily as one written last week. Algorithms still can’t craft a narrative the way a person can — they can’t write a decent screenplay, or pass Schank’s Romeo and Juliet test. For the most part, they can’t reason about cause and effect. They can’t write stirring prose, and they can’t persuade a public official to go on the record about an important policy. Still, there’s plenty they can do. AI may not be able to tell a great story, but it can help us better tell our own. The Big Idea

Why AI Can’t Write This Article (Yet)

Partner Center