The biggest names in artificial intelligence have agreed to honor a single signal that lets publishers block AI scraping across several crawlers at once. OpenAI, Google, Microsoft and Anthropic said they will adopt the shared standard, first reported by Search Engine Roundtable and detailed by The Verge, so a website can apply its AI training preferences in one place instead of negotiating with each company individually.
The move lands as news organizations and independent sites struggle with a growing wave of AI crawlers that pull their content into model training. Until now, blocking those crawlers meant editing robots.txt for one provider at a time, a slow and error-prone job for small editorial teams. The new standard is meant to turn that into a single switch.
How the AI opt-out standard works
The mechanism is a shared "policy token" proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF). It lives inside a website's robots.txt file, the decades-old text file that already tells crawlers whether they may index or archive a site. Once a publisher places the token there, participating AI companies say they will restrict their crawlers from accessing the site.
That is a change from the current patchwork, where each AI company offers its own controls and publishers must repeat the same opt-out over and over. Microsoft added that it will offer granular controls, letting publishers separate AI training from AI grounding, the process of pointing a model to a verified source to reduce hallucinations. Google already has similar controls covering AI search features and AI training without affecting search visibility.
- Anthropic has already adopted the signal, according to The Verge.
- Google and Microsoft plan to support it next year.
