Google is set to release a code for an open-source anti-harassment tool called Harassment Manager. The tool is geared at helping women journalists, public figures and activists, especially those covering controversial topics or living under autocratic governments, manage online abuse. The tool employs Jigsaw’s Perspective API and gives users the ability to sort through potentially abusive comments via social media platforms starting with Twitter. It’s debuting as source code for developers to build on, then being launched as a functional application for Thomson Reuters Foundation journalists in June.
According to reports, the Harassment Manager is known to work effectively with Twitter’s API to combine moderation options. These moderation options include hiding tweet replies, muting or blocking accounts, also with a bulk filtering and reporting system. The software is known to check for levels of “toxicity” in online messages’ based on elements like threats, insults, and profanity. Reports say it sorts messages into queues on a dashboard and allows the user to address them in batches rather than one after the other through Twitter’s default moderation tools. Also, users have the ability to blur such messages while moderating, so they don’t have to read, and can also perform a search using keywords in addition to using the automatically generated queues.
In addition to the multiple benefits the new Harassment Manager Tool presents, users can also download a standalone report that contains all the abusive messages. Reports say this creates this will be effective in the creation of a paper trail for their employer or, in the case of illegal content like direct threats, law enforcement or other needs that might arise. For now, however, there’s not a standalone application that users can download. Instead, developers can freely build apps that incorporate its functionality and services using it will be launched by partners like the Thomson Reuters Foundation.
The Harassment Manager particularly relevant to female journalists who face gender-based abuse was announced today which significantly is International Women’s Day. The “journalists and activists with large Twitter presences” as well as the International Women’s Media Foundation and the Committee To Protect Journalists have collectively via a Medium post said that it’s hoped developers can tailor it for other social media users that might be at risk. According to the team “Our hope is that this technology provides a resource for people who are facing harassment online, especially female journalists, activists, politicians and other public figures, who deal with disproportionately high toxicity online.”
This is not the first time Google has harnessed these platforms to deliver automated moderation. In 2019 a browser extension called Tune was introduced, this browser extension was responsible for helping social media users avoid coming in contact with messages with a high chance of being toxic. The browser extension is said to have been used by several commenting platforms (including Vox Media’s Coral) to supplement human moderation. But as we noted around the release of Tune, the language analysis model has a history of showcasing errors which are suggested signs of it being far from perfect. It sometimes misclassifies satirical content or fails to detect abusive messages, and Jigsaw-style AI can inadvertently associate terms like “blind” or “deaf” — which aren’t necessarily negative — with toxicity. Jigsaw has received critics for practices that showcase some form of toxic workplace culture, although Google has disputed the claims.
Unlike other AI-powered moderation services available on Twitter and Instagram which you might have come across, Harassment Manager isn’t a platform-side moderation feature. It’s apparently a sorting tool for helping manage the sometimes overwhelming scale of social media feedback, something that could be relevant for people far outside the realm of journalism — even if they can’t use it for now.
