Why it matters: Instead of writing separate rules for each AI company's crawler, a site could state one training preference that every participating provider reads through the shared signal, while OpenAI and Google say their existing controls stay in place.
OpenAI said on Oct 30 that it will honor a common signal in the robots file that lets websites state whether their content may be used for foundational generative AI training. The robots file is the plain-text file at the root of a website that sites use to give instructions to automated crawlers. OpenAI says the signal lets a site send the same preference to several participating AI providers at once, instead of writing separate rules for each company's crawler.
Google is joining: the same day, Google said it plans to support the shared AI training control. Google says that when a site uses the shared token to opt out of training and does not name Google-Extended, Google plans to treat it as if the site had disallowed Google-Extended, the AI training control Google says it has offered since 2023.
How the signal works: the mechanism is set out in an Internet-Draft dated Oct 30, 2026 for the AI Preferences working group of the Internet Engineering Task Force (IETF). Its listed author is at Anthropic. The draft defines a shared policy token, usage_ai-training with two leading underscores, that goes in a User-Agent line of the robots file.
- Opting out: a Disallow rule covering the whole site under that token expresses a preference against AI training for the whole site, and Allow rules express a preference in favor for those paths.
- Not a crawl block: the draft says the token is not a directive for crawling behavior. Sites that want to limit crawling still use rules for each crawler.
- Status: an Internet-Draft is a working document, not a finished standard. This one expires on May 3, 2027 unless it is updated.
When it takes effect: neither company reads the signal yet. OpenAI says it aims to implement it in the coming months and will share details when it does. Google says it aims to support the control along with other AI providers in 2027.
What stays the same: OpenAI says its existing crawler controls will continue to apply, and Google says company-specific controls such as Google-Extended keep working alongside the shared signal. Google also says the proposal is consistent with the EU's Code of Practice for general-purpose AI model providers, which it says recognizes the robots file as a main way for publishers to state machine-readable preferences.
Sources
- OpenAI: "Making it easier for websites to express AI training preferences," Oct 30, 2026
- Google Search Central Blog: "Supporting a shared AI training control for websites," Oct 30, 2026
- IETF Internet-Draft: "Machine-Readable Training Preference Signal" (draft-kelley-aipref-training-preference-signal-00), Oct 30, 2026
How we checked this: We read OpenAI's Oct 30, 2026 announcement, Google's Search Central blog post of the same day and the IETF Internet-Draft and its datatracker page, in copies saved on Oct 31, 2026 at about 7:57 AM ET, and checked every date, name and technical detail against them.
See something wrong? Tell us and we'll fix it. Every fix is dated and listed on our Corrections page.