Key points
- Google, Microsoft and OpenAI said Oct. 30 they intend to respect one common robots.txt rule for AI training, drawn from an Internet-Draft by Anthropic's Kevin Kelley.
- None of the three supports the rule yet, per their announced timelines; Microsoft stops treating NoArchive and NoCache as AI preferences after March 1, 2027.
- Opting out through the shared group would not cover identical uses at Google and Microsoft, since Google-Extended also covers Gemini Apps grounding.
Why it matters for marketers: Sites could manage AI training permissions in one robots.txt group, but marketers still need per-company checks because the opt-out will not mean the same thing everywhere.
Google, Microsoft and OpenAI announced on Oct. 30 that they intend to respect a single robots.txt rule that lets site owners state whether their content may be used for AI training. The rule comes from an Internet-Draft written by Kevin Kelley of Anthropic. It works as an ordinary robots.txt group tied to a policy token called __usage_ai-training, and it accepts both Allow and Disallow lines.
Right now, none of the three companies supports it, based on the timelines they have published. Google said that when a site has not set Google-Extended, it will read the shared opt-out as if the site had opted out of Google-Extended. Microsoft plans to apply the shared rule to training, add a separate page-level AI Grounding preference, and stop treating NoArchive and NoCache as training or grounding signals after March 1, 2027.
For content teams, the appeal is obvious: one group could replace a pile of per-company training rules. The catch is scope. Google's documentation says Google-Extended also covers grounding in Gemini Apps, and it does not separate that from Gemini training. So an opt-out would not cover the same uses at Google and Microsoft.
Teams that rely on NoArchive or NoCache with Microsoft have until March 1, 2027 to move to the new preferences. Until the companies start reading the shared rule, keep your existing crawler-specific rules in place. When support arrives, document exactly what each company says the opt-out covers before you update your robots.txt policy.
Sources
This brief is our own summary. Read the full reporting at the source.