One Robots.txt Rule for Every AI Crawler: OpenAI, Google and Microsoft Back a Shared AI Training Opt-Out Signal
📑 Table of Contents
The Whack-a-Mole Problem With AI Crawlers
For three years, blocking AI companies from training on your content has meant playing whack-a-mole in your robots.txt file. Every provider ships its own crawler with its own name — GPTBot for ChatGPT, ClaudeBot for Claude, Google-Extended for Gemini, CCBot, Bytespider, PerplexityBot and a growing list of others. Each one needs its own rule, each rule needs to be kept up to date as new crawlers appear, and a site that misses one silently feeds yet another model.
On October 30, that fragmentation took a real step toward ending. OpenAI, Google and Microsoft each announced they will honor a single shared robots.txt signal that tells every participating AI provider the same thing: don't train on this site. Anthropic, according to The Verge, had already adopted the signal ahead of the announcement.
It sounds like a boring technical detail. It isn't. It's the first time the major AI labs have agreed on a common machine-readable way for publishers to say no — and it changes the calculus for anyone who runs a website, writes a blog, or ships an AI product that touches other people's content.
What the Shared Signal Actually Is
The mechanism is a common "policy token" placed in a site's robots.txt file. Instead of enumerating individual crawler user-agents, a site operator declares a preference once, and every provider that supports the token reads the same instruction.
The groundwork comes from the IETF's AI Preferences working group, the standards body that has been hammering out how websites should express machine-readable preferences to AI systems. On the same day as the announcements, Anthropic submitted an Internet-Draft describing the signal, moving it down the formal standardization track.
Until now, robots.txt could only speak crawler-by-crawler. The new token is designed to be the opposite: one entry, read by many. Company-specific controls aren't going away — they layer on top, so sites that want fine-grained rules (allow one provider, block the rest) can still write them.
Who Implements It, and When
The commitments are real but staggered, and no one flips the switch today:
| Company | Commitment | Timing |
|---|---|---|
| Anthropic | Already honoring the signal | Live now |
| Microsoft | Honors new preferences as it crawls | From February 1, 2027 |
| Supports alongside other providers | Targeting 2027 | |
| OpenAI | Implementation announced | "Coming months" |
Google also spelled out an important compatibility detail: when a site uses the shared token without an explicit Google-Extended rule, Google will read the opt-out as a disallow of Google-Extended — its existing control for training and grounding use of public web content.
What It Covers — and What It Doesn't
The headline is "AI training opt-out," but the details matter for anyone who publishes online:
- Training vs. grounding are different things. Microsoft is adding a separate, page-level preference for grounding — the use of web content to inform generated answers — and says sites remain eligible for Bing search results either way. Opting out of training does not automatically opt you out of being quoted in an AI answer.
- It's opt-out, not opt-in. The default remains: crawlers may train unless you say no. Publishers who wanted a licensing-first internet didn't get that here.
- Compliance is voluntary. The signal binds the companies that pledged to honor it. Crawlers from providers that didn't sign up can still ignore it — which is why existing per-crawler blocks still matter.
- Regulatory tailwind. Google noted the approach is consistent with the EU AI Act's Code of Practice for general-purpose AI model providers, which treats robots.txt as a primary way for publishers to express machine-readable preferences. In Europe, honoring these signals is edging from courtesy toward obligation.
✅ What improves
- One rule reaches multiple major AI providers
- Standards-track process at the IETF, not a proprietary format
- Company-specific controls remain for fine-grained policies
- Aligned with EU AI Act expectations
❌ What doesn't
- Rollout stretches into 2027 for Google and Microsoft
- Non-participating crawlers can still scrape
- Training opt-out doesn't cover AI-answer grounding everywhere
- Enforcement still rests on each company's good faith
Why This Is Happening Now
The pressure has been building for a while. Publishers have spent 2026 fighting AI summarizers that absorb their traffic, CDNs like Cloudflare shipped aggressive default AI-bot blocks for their customers, and licensing deals became the preferred path for the biggest news organizations. Regulators piled on: the EU AI Act's Code of Practice explicitly leans on robots.txt as a publisher-preference mechanism, and scrutiny of scraping practices on both sides of the Atlantic has only intensified.
For the AI companies, a fragmented, per-crawler status quo was also a liability — every new crawler name that appeared in a robots.txt file was another news cycle about "the bot you didn't know about." A single, standardized, opt-out-first signal is cheaper for them and more defensible in front of regulators, even if it means respecting more noes.
What Website Owners Should Do Today
Practically, not much changes this week — but a little preparation pays off:
- Keep your existing per-crawler blocks. They're the only rules fully enforced today, and they remain the fallback for non-participating crawlers.
- Watch the IETF draft. The final token syntax is still moving through standardization; copying a placeholder from a blog post (including this one) before the draft settles can leave you with a rule nobody reads.
- Audit what you actually want. Decide separately for training vs. grounding vs. search indexing — Microsoft's page-level grounding preference means these can diverge.
- Check your CDN defaults. If you're behind a provider with AI-bot controls, your effective policy may already be stricter than your robots.txt.
What It Means If You Build With AI Tools
If you're using AI coding agents like Claude Code or Gemini CLI, or shipping products on top of models from DeepSeek or Grok, this trend lands on you too. Agentic tools that fetch live web pages increasingly inherit the obligation to respect publisher preferences — GitHub Universe 2026 previewed code scanning that flags indirect prompt injection and excessive agency in AI apps, and preference-respecting crawling is the natural next frontier. Treating robots.txt and preference signals as first-class input, rather than an afterthought, is becoming table stakes for AI tooling.
Frequently Asked Questions
Does the new signal replace blocking GPTBot and other crawlers individually?
No. Company-specific controls stay in place and remain the only fully enforced mechanism today. The shared token is an additional layer that lets one rule reach every participating provider at once.
If I opt out of AI training, will AI chatbots stop quoting my site?
Not necessarily. Training opt-out and grounding (using content to inform generated answers) are handled separately. Microsoft, for example, is introducing a page-level preference for grounding, and sites that opt out of training remain eligible for search results.
When can I actually add the signal to my robots.txt?
Wait for the published syntax from the IETF AI Preferences working group process and the first provider implementation notes. OpenAI expects to implement "in the coming months"; Microsoft starts honoring preferences February 1, 2027; Google targets 2027.
Is this legally enforceable?
Not on its own — it's a voluntary commitment. But it aligns with the EU AI Act's Code of Practice for general-purpose AI providers, which treats robots.txt preferences as a primary mechanism for publishers, giving the signal regulatory weight in Europe.
Build With AI Tools That Respect the Rules
Discover and compare 1,100+ AI tools on aitrove.ai — your trusted AI tool directory.
Browse All Tools →