Cloudflare Splits AI Bots Three Ways: How New Search, Agent & Training Controls Change the AI Tools You Use in 2026

The News: Content Independence Day, Round Two

On July 1, 2026, Cloudflare declared its second annual Content Independence Day, and this time the company is retiring the blunt instrument it became famous for. A year ago, Cloudflare handed website owners a one-click "Block AI Bots" switch alongside a Pay-Per-Crawl marketplace — a direct response to AI companies that were "taking everything and sending back nothing." That blunt block was a powerful statement, but customers quickly wanted more nuance.

The new release, detailed in a blog post by Cloudflare's Jin-Hee Lee and Bryan Becker, replaces the single-purpose "training only" block with finer-grained controls available to all customers, including those on the Free tier. Instead of asking "is this bot AI or not?", Cloudflare now asks a more useful set of questions: What is the bot doing on my site? What is it storing? And how will it reshare my content? The answers map onto three distinct use cases — Search, Agent, and Training — that site owners can now manage independently.

The Problem: A 30-Year Deal That Stopped Working

For three decades, the web ran on an implicit bargain: a crawler indexes your pages, and in return you get referral traffic. AI broke that bargain. Answer engines now read your content and answer the user directly on the results page, sending nothing back. That left publishers with what Cloudflare calls a Faustian bargain — either show up in search and let AI train on you for free, or block the bots and risk vanishing from discovery entirely.

The dilemma unfairly advantages incumbent search providers that reuse the same crawlers for both search and training, while incentivizing newer entrants to be evasive to close the competitive gap. Cloudflare's argument is that the binary "block everything" lever was never the right shape for a world in which "AI can be in anything" — where Google is simultaneously a search engine, an answer engine, and a model trainer, often through overlapping automation.

The New Taxonomy: Search vs. Agent vs. Training

The centerpiece of the update is a pragmatic three-way classification of AI-centered bot behavior:

• Search — crawls or indexes your content so it can answer questions about it later. It is proactively building a database of your site, and the expectation is that it funnels referral traffic (or other equitable compensation) back to you.
• Agent — automated behavior acting in real time on a person's behalf to get something done right now. Cloudflare explicitly calls out chat-fetch bots like ChatGPT-User and browser-use agents like Gemini or Claude driving Chrome. A human is usually waiting on the other end.
• Training — a crawler taking your content to train or fine-tune a model, where your data is permanently absorbed into the model's architecture.

Crucially, a single bot can carry more than one label. Cloudflare now tracks all of a bot's purposes rather than forcing a single tag, and it is pressuring bot operators to split their automation into separate, clearly named crawlers so site owners can actually tell what is happening. Alongside the three AI use cases, the broader taxonomy also covers Transact, Data Collection, Security Testing, SEO, Ads Verification, Social/Link Preview, Feed Fetching, and Monitoring & Operations.

The September 15 Defaults (and the Googlebot Trap)

The most consequential change is the new default policy taking effect on September 15, 2026. For all new domains onboarding to Cloudflare, the Training and Agent categories will be blocked by default on pages that display ads, while Search remains allowed by default. The logic: an ad signals that a human was meant to land on that page and see it, so the bots that prevent that human attention — Training and Agent bots — get locked out.

Because defaults are enforced by the most restrictive applicable rule, multi-purpose crawlers like Googlebot, Applebot, and BingBot (which combine Search and Training) will be blocked for any customer who blocks Training — through either the new options or the legacy Block AI Bots service.

Existing customers who want to keep things exactly as they are today can opt out in their Security settings any time before September 15. Cloudflare says it will keep notifying customers as the date approaches. The company is also launching BotBase, a new visibility database for Enterprise Bot Management that shows a comprehensive, searchable catalogue of all Verified bots and agents and how each is classified — with direct control capabilities planned to follow later in the year.

The use= Robots.txt Signal, Explained

Granular classification only helps if site owners can also express how much of their content a bot may keep and reshare. To that end, Cloudflare is rolling out a "content use" setting for Bot Management with three levels, from least to most permissive:

• immediate — interact, but store and reuse nothing
• reference (default) — index, excerpt, and link back
• full — summarize and reproduce

These map onto a brand-new signal that lives in your robots.txt: use=immediate, use=reference, or use=full. It extends Cloudflare's Content Signals (now a fourth, optional field) and lets you express rules like "allow Search, SEO, and Ads Verification bots, but only up to the reference level." For anyone shipping an AI agent or answer engine, that signal is now a first-class part of the crawl negotiation.

What It Means for the AI Tools You Use

If you are choosing AI tools in 2026 — whether you run a website that feeds them or you build on top of them — Cloudflare's shift changes the calculus in concrete ways:

• Expect AI search tools to behave differently on ad-heavy pages. From September 15, Agent and Training bots are off by default where ads appear. Tools that blur the line between "answer this question" and "train on this page" will increasingly get blocked unless their operators separate their crawlers.
• Browser-use agents are now a managed category. Gemini or Claude driving Chrome, and fetch bots like ChatGPT-User, are explicitly classified as Agent traffic. Sites can welcome, rate-limit, or block them independently of search crawlers — so the agent tools you pick need to respect robots.txt and the new use= signal, or they will be shown the door.
• Multi-purpose crawlers carry the most risk. Because defaults apply the most restrictive rule, a bot that crawls for both Search and Training inherits the Training block. Vendors that refuse to split their automation will lose access to a growing slice of the web.
• Publishers gain a real negotiating layer. Combined with Pay-Per-Crawl and the content-use tiers, site owners can finally trade access for compensation instead of choosing between free exploitation and total invisibility.

The throughline: the era of "block all AI bots or block none" is ending. The tools that win will be the ones that tell the truth about what they are doing — Search, Agent, or Training — and let the web decide on those terms. Explore the AI Search & Chat tools and AI Agent & Coding platforms on aitrove.ai to compare options that play by the new rules.

Frequently Asked Questions