Key points
- OpenAI, Google and Microsoft plan to honor a shared robots.txt token for AI training opt-outs; Anthropic's Kevin Kelley wrote the IETF draft and says Anthropic has started building support.
- Google targets early 2027; Microsoft starts honoring it February 1, 2027; OpenAI says it will implement it in the coming months.
- Microsoft is replacing NoArchive and NoCache for training and grounding, and a temporary Bing Webmaster Tools setting opens Dec. 15 for sites still using those tags.
- The draft is an individual IETF submission, not official standardization, and a missing token group means no preference.
Why it matters for marketers: Marketers managing site access could write one training opt-out instead of per-company rules, but Microsoft's tag changes and Google's bundled controls mean current settings need review before 2027.
Several major AI companies have said they will honor a shared robots.txt signal that lets a site state its AI training preference once, rather than writing separate rules for each company's crawler. The proposal comes from Kevin Kelley of Anthropic, who submitted it as an individual draft to the IETF's AI Preferences working group. Search Engine Journal notes that this status does not represent official standardization.
The token adds no new directive. Sites would use ordinary Allow and Disallow lines under a group named __usage_ai-training. A wildcard group can never match it, and a site without the group is treated as expressing no preference. It does not control crawling.
Timing varies. Anthropic has started development, OpenAI says it will implement the signal in the coming months, and Google plans support in early 2027. Microsoft will begin honoring it on February 1, 2027. Meta said a common signal is an important step but did not say whether it will honor it.
The details differ by company. Google says it will treat the token like a Google-Extended block when that setting is absent, which also covers grounding in Gemini Apps. Microsoft treats training and grounding as separate preferences, and says restricting either can reduce brand visibility in AI answers without affecting Bing search listings.
Teams that use NoArchive or NoCache tags to limit Copilot should plan now. Microsoft will stop recognizing those tags for training and grounding after March 1, 2027, and a temporary Bing Webmaster Tools setting opens Dec. 15. Also watch for how OpenAI handles conflicts between the shared token and its GPTBot rules, which its documentation did not address.
Sources
This brief is our own summary. Read the full reporting at the source.
