How To Stop Online Harassment: Google Uses Machine Learning Tools To More Accurately Spot Abusive Content

A subsidiary of Google’s parent company Alphabet, Jigsaw, is using machine learning to fend off online trolling, reports Wired. The New York–based think tank is building open-source AI tools, collectively called Conversation AI, to filter out harassment and abusive language.

“Few things poison conversations online more than abusive language, threats, and harassment,” reads the Conversation AI website. “We’re studying how computers can learn to understand the nuances and context of abusive language at scale. If successful, machine learning could help publishers and moderators improve comments on their platforms and enhance the exchange of ideas on the internet.”

Conversation AI identifies language that could potentially be abusive and gives it an “attack score” that can range from 0 to 100—the higher the score, the more problematic the language. The software is reportedly faster and more accurate at identifying abusive language than human moderators. According to Google, Conversation AI is “more than 92 percent certainty and a 10 percent false-positive rate” but Wired ’s Andy Greenberg found some room for improvement.

“I want to use the best technology we have at our disposal to begin to take on trolling and other nefarious tactics that give hostile voices disproportionate weight,” Jigsaw founder and president Jared Cohen told Wired . “To do everything we can to level the playing field.”

To create the tool, Jigsaw partnered with The New York Times and the Wikimedia Foundation. The company got access to 17 million comments from stories produced by Times and data on which comments were flagged by human moderators to be inappropriate. It also received 130,000 pieces of conversations from Wikipedia pages, which were then given to a human panel for evaluation.

The Times and Wikimedia will be the first to test out the abuse detector. While the latter is still looking at how it can best utilize the tool, the former will be implementing Conversation AI as the first line of defense to weed out abusive content before human moderation.

By making the information open-source, any forum or platform can use the software to flag, delete and moderate online trolling. Jigsaw Product Manager CJ Adams told Wired the goal is for “anyone can take these models and run with them.”

The significance of Conversation AI is transparent: it could make it easier to stop online harassment. But a lesser known potential perk is that, if effective, it can transform online behavior. Previous research, published in the journal Nature, has found that when called out for abusive comments, people changed their behavior. Specifically, video game company Riot Games found that 92 percent of players modified their behavior after receiving real time automated warnings.