OpenAI and Google say they will honour a shared robots.txt signal for AI training

OpenAI's post, dated 30 October 2026, says it will honour a common signal in robots.txt for the use of a site's content for foundational generative AI training. It calls the step voluntary. It says the signal will not replace OpenAI's existing robots.txt controls, which will continue to apply. It says it aims to implement the signal in the coming months and will post again when it does. It says the move draws on the IETF AI Preferences working group, and that OpenAI is joining other participants in that group's announcement the same day. The post does not name the token and does not name the other companies.

Google's Search Central post, dated Friday 30 October 2026, says Google plans to support a shared AI training control in robots.txt, based on the same working group. It says a site will be able to use a shared policy token. It says that if a site uses that token to opt out of training, and Google-Extended is not explicitly specified, Google plans to treat the opt-out as if the publisher had disallowed Google-Extended. It says existing company-specific controls will continue to work alongside the shared signal. It says Google aims to support the control, along with other AI providers, in 2027. It also says the proposal is consistent with the EU Code of Practice for general-purpose AI model providers, which it says recognises robots.txt as a primary way for publishers to express machine-readable preferences.

An individual Internet-Draft on the IETF datatracker, dated 30 October 2026, describes the token as __usage_ai-training in the User-Agent line of an ordinary robots.txt group. The datatracker says an individual Internet-Draft is not endorsed by the IETF and has no formal standing in the IETF standards process. The draft lists its author affiliation as Anthropic. That is not an Anthropic newsroom post saying Anthropic will honour the signal. Anthropic's news index, opened the same day, had no item dated 30 or 31 October.

What it means for business: neither company says the shared signal is in force on its crawlers today. OpenAI's timing is "the coming months". Google's timing is 2027. A site that already blocks a named crawler is not told to remove that rule. The draft is work in progress, not a standard.