SXSW 2015: Robots, Algorithms Infiltrate The News Business, Selecting Stories And Writing Them

AUSTIN, Texas -- Three years ago, media companies told Automated Insights CEO Robbie Allen that they didn’t think it was possible to have robots or software write news articles. On Sunday, Allen predicted that every single major news organization will roll out at least one kind of automated product by the end of the year “just to stay relevant.”

“It’ll become pervasive,” Allen said.

Allen delivered his remarks during “When Robots Write the News, What Will Humans Do?” -- a conversation between Allen and Lou Ferrara, the vice president of sports, entertainment and business news at the Associated Press, who has worked together with Allen to bring automation into certain parts of his vast news organization. Last July, the two companies announced that Wordsmith, a program developed by Automated Insights, would be working with data shaped by Zacks Investment Research to write quarterly earnings stories. Earlier this month, the companies announced they would work with NCAA data to begin writing certain kinds of college baseball and basketball recaps next year.

Both emphasized that the steps ahead in the move toward automation in the news business are numerous. "We're still at the infancy of this," Ferrara said, though he added he expects to build quickly on the foundation that’s been laid. "By the end of the year, we want to come away with three or four types of automation that are clear wins," Ferrara said.

At the moment, the kinds of stories that Wordsmith can write are limited. It is designed to draw on what is called structured data, or information that is presented in a consistent, formalized way, like formatted quarterly earnings data, or a box score from a sporting event. Wordsmith is not designed to do things such as summarize speeches or review movies or explain what happened at a crime scene.

Yet, even in this limited context, Wordsmith offers the AP’s newsroom staff a reprieve from one of the most frustrating kinds of work. The earnings story is in many ways a relic of an earlier era of the news business, one which, Ferrara says, reduces his reporters to “data processors.”

Ferrara told panel attendees that Wordsmith freed up 20 percent more time for his business reporters to focus on other kinds of work, thereby offering readers and AP clients a higher-quality product.

“The bar is going to continue to get raised in this business,” Ferrara said of automation.

Influencing Editors

But automation’s influence already extends well past the writing process. At a separate panel, “Data-Driven Strategy Versus Editorial Gut Instinct,” panelists discussed how many top media brands are already using automated digital tools to shape their editorial strategy and thinking: Dataminr surfaces topics gathering popularity on social media, CrowdTangle highlights what’s working on sites run by competitors, and Parse.ly unpacks how certain pieces of content perform in certain contexts.

Though many of these companies are barely a couple years old, they are already standard tools of the trade. CrowdTangle, still in private beta, is already in use by most of the country’s top publishers, because it offers them easily digestible ways to maximize profits with their content. "Every editor now has to be a business strategist every single day," Danica Lo, the digital editor-at-large for Glamour.com and the former executive editor of Epicurious, said.

Where once editors simply used tools like Google Analytics to track how well sites or pages performed over the course of a week or a month, they are now expected to win news cycles that last just hours by delivering content they know is going to hit big on social networks and among the site's readership.

That’s where the robots come in.

“The conversation about data is happening earlier in the editorial process,” Elizabeth White, the head of strategy at Poshly and the former vice president and general manager of Entertainment Weekly’s website, said. “When you have limited resources, it's tempting to turn over your editorial strategy to an algorithm,” she added.

Sea Of Sameness

Poring carefully over the data these tools surface can sometimes lead editors to unlikely mother lodes of traffic. Jessica Novak, the content strategist at Refinery29, credited careful data analysis for her publication’s decision to put a huge amount of editorial energy into covering Kendall Jenner, back before the young model had become a high-profile figure.

Editors and data experts at R29 noticed posts about Jenner always outperformed expectations, creating a positive feedback loop that wound up brightening Jenner’s star and providing another reliable source of good traffic for the site. "Now we've integrated her into all of our categories," Novak said.

Yet these tools are not always used so carefully. Many editors use them superficially or without a great deal of rigor. And because so many media companies use the same ones, they often wind up pouncing on the same stories at the same time, leading to nearly instant saturation. “I needed to shut down my feeds when #TheDress happened,” Poshly’s White said of a notorious recent example.

Because of the effectiveness of the tools, panelists Sunday expect that their influence will only grow. Many of them hope that as they grow, the level of sophistication brought to bear on them increases as well. “I would love to see publishers looking more for outliers as part of their programming strategy,” White said. “That's what the data is for.”