Meta Clarifies Crawling Terms After Publishers Raise AI Alarm

Meta has revised language in its commercial terms after publishers raised alarms that a broad crawling clause could be read as permission to collect their websites for AI training. The October 17 change followed Nieman Journalism Lab’s examination of the draft terms with legal experts. Meta says the disputed wording was meant for its ads and business products, and that the clarification does not change how its web crawlers operate.

This is a dispute about the reach of a contract, not evidence that Meta newly scraped a particular newsroom or trained a model on its work. That distinction matters. A publisher can object to language that appears to grant sweeping rights even before anyone proves that those rights have been used. The episode shows how difficult it has become for newsrooms to separate platform access from control over the content on their own sites.

The proposed terms, due to take effect on October 30, covered websites associated with a business account and described the use of automated tools to access and collect material. They also said content from those domains could be used to develop, improve, research or train Meta products. Legal experts consulted by Nieman Lab argued that the wording could encompass publishers that rely on Meta business accounts to manage Facebook, Instagram or WhatsApp activity.

The size of that potential category is what made the clause important. A local publisher may use Meta products for audience reach and advertising without thinking it is signing a separate licence for the contents of its news website. The original wording also raised questions about how contractual permission would interact with a site’s robots.txt instructions, which tell crawlers what they may access. Nieman Lab reported that Meta’s spokesperson said content owners could still use robots.txt and that the change did not alter crawling practice.

Subsequent reporting on Meta’s revision said the disputed provision was removed. Meta’s explanation was narrower than an admission of a new scraping policy; the company said it had consolidated wording that applied to ads and business products and clarified that scope in the terms. Publishers will still want to inspect the effective text and any supplemental terms rather than rely only on a general reassurance.

The wider AI publishing problem remains

Generative AI has increased the value of large collections of reliable writing, while the platforms that distribute news are also among the companies building AI systems. That creates a structural tension for publishers. They need discovery and referral traffic, yet want a meaningful say over whether their reporting is used to train or power competing products. TechBooky has covered Cloudflare’s effort to give sites more control over AI crawlers, one technical response to that tension.

A contract amendment is only one layer of the problem. Newsrooms also need to know which crawlers visit their sites, how the data is used, whether opt-outs are honoured and what compensation, if any, is available. Platforms should make those answers intelligible to a small publisher as well as to a legal department. Meta’s edit reduces the immediate ambiguity that triggered criticism, but it does not settle the broader argument over who controls the value of published work in the AI era.