Cloudflare spent a year trying to split Googlebot. It settled for a line in robots.txt

On 15 September the company retired its one-click AI-blocking switch. What replaced it lets Google, Apple and Microsoft keep crawling, on a promise. Every other company's training bots stay locked out, and one name is missing from the announcement altogether.

Engines · 16 September 2026

Somewhere in your inbox this week, if your site sits behind Cloudflare, is an email with the subject line "Updates to managing AI crawlers on your account". It is written the way these emails always are, calm and administrative, and it says that most customers won't need to change anything [Verified, S1]. That is true. It is also the least interesting thing about it.

Here is what the email describes. The Block AI Bots switch, the one-click control Cloudflare launched in July 2024 and that more than a million customers had turned on within a year [Claimed, S21], is gone. In its place are three separate controls, one each for Search, Training and Agent crawlers, and a new Training option called Disallow AI Training. If your switch was on, your site has already been moved: Search set to Allow, Training set to Disallow AI Training, Agent set to Block on pages with ads [Verified, S1, S2]. You did not have to click anything. It happened on Tuesday, and the dashboard will catch up over the week [Verified, S1]. The full mapping is short enough to print.

Table 1: How the old switch was migrated

What the email does not say is that this is not the plan Cloudflare announced in July. Then, it said that from 15 September any site that blocked AI training would block Googlebot, Applebot and Bingbot too, because those three collect for search and for AI training in a single visit, and Cloudflare was no longer willing to let them have it both ways [Verified, S3]. Eleven weeks later, the three crawlers are in. What keeps them honest is a line in the site's robots.txt asking them not to train, which Google and Apple say they honour and Microsoft says it will honour from early 2027 [Verified, S2].

So this is a story about a threat that was withdrawn, and about the shape of what replaced it. It matters to anyone who wants to be named when a person asks ChatGPT, Gemini, Perplexity or Copilot a question, because the settings in that email decide which of their crawlers can still read your pages. And it has a wrinkle that took us a while to notice: the setting called Disallow AI Training also switches off something that is not training.

The fence and the sign

Cloudflare's whole pitch on AI crawlers, from the start, has been that it is not robots.txt. Robots.txt is a sign on the lawn. Anyone can put one up and nobody has to obey it. Cloudflare sits in front of more than a fifth of the web's domains [Claimed, S3] and can refuse the connection. That is a fence. The company said as much again on Tuesday, under a heading that reads "Why asking isn't enough": a robots.txt file "cannot identify who is crawling, determine why they are crawling, or stop a crawler that ignores it" [Verified, S2].

Which is what makes the new setting worth reading slowly. Disallow AI Training, in Cloudflare's own words, "works by publishing a preference in robots.txt" [Verified, S2]. The publishing is done by Bot Preference Sync, a feature launched on 21 August that writes managed lines at the top of a site's existing robots.txt [Verified, S4]. For Googlebot, Applebot and Bingbot, the fence opens and a sign goes up. Every other training crawler is still blocked at the edge, "including the training-only crawlers run by Amazon, Anthropic, Meta, and OpenAI", which, as the post points out, costs a site nothing in search because those companies run separate crawlers for it [Verified, S2].

Cloudflare has a word for the three it trusts: accountable. To earn it, an operator must meet, or commit to meeting, four conditions. A way to opt out of training through robots.txt or something like it. A way to opt out of AI summaries, "set with the operator directly, and next year through Cloudflare". URL-level visibility into which pages were made available for training, plus search metrics. And an assurance that opting out of training "will not affect traditional search results" [Verified, S2]. Apple, Google and Microsoft are the three named, and Cloudflare's summary of where they stand is worded with care: they "honor or have committed (in a specified time frame) to honor this setting" [Verified, S2].

Look at who is on that list. In January Cloudflare published a policy paper arguing that Google's combined crawler gave it an unfair advantage and asked regulators to force a split [Verified, S13]. In September, the operators whose combined crawlers it objected to are the ones whose crawlers keep flowing. That is our observation, not Cloudflare's [Inferred (ours)].

A year of saying no

To see how the company got here, go back to 1 July 2025. That was the day Cloudflare announced it was "changing the default to block AI crawlers unless they pay creators", with Matthew Prince arguing that "content is the fuel that powers AI engines" and that the people who make it should be paid [Verified, S10].

Google was the problem from the beginning, because Google is the one company no publisher can afford to block. In early December Prince said so in public, at a WIRED event: "Google is the problem here", and Google "should split their crawlers up between search and AI" [Claimed via press, S11]. In January the company put it in writing. A framework "where the platform dictates the rules", it wrote, does not offer publishers "effective control" [Verified, S13]. In February it took that argument to the UK's Competition and Markets Authority, which had given Google strategic market status the previous October, and asked for a rule "requiring Google to separate traditional search indexing from AI training and grounding crawlers" [Verified, S12].

The CMA did not give it that. Its final decision, on 3 June, requires Google to let publishers opt out of AI Overviews and AI Mode and out of fine-tuning, with nine months to comply and reports every six months for a year. It does not mention splitting the crawler [Verified, S14, S15]. Google built its opt-out as a toggle in Search Console, tested it with UK sites from June and opened it to every site on 31 August [Verified, S17, S18].

So when Cloudflare's anniversary post landed on 1 July 2026, it read like a company deciding to do on its own what the regulator would not. It announced the three categories, and then the sentence that set the tone for the summer: "Multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training", through the new controls or the old switch, from 15 September [Verified, S3]. The SEO trade read it as a countdown [S8, S9]. On 4 August, Search Engine Journal picked up a Reddit thread in which a site owner said that setting Training to Block was already sending 403 errors to Googlebot and Bingbot. Google's John Mueller asked the poster to get in touch. Nobody confirmed it either way [Verified that the report exists, S7].

Then the date arrived and the block did not. Cloudflare's post does not present this as a change of course. What it says is that "since July, we've been talking to them directly", and that "almost all agreed that site owners should have control and transparency" [Verified, S2]. Almost all. It does not say who the exceptions were [not disclosed].

What a line in a text file does

For the three companies the setting is not one thing. It is three different things, and the differences are the point.

Start with Google. Cloudflare writes a Disallow line for Google-Extended [Verified, S2]. Google's documentation for that token, last updated on 14 July, says it controls whether a site's content is used to train future Gemini models, and also whether it is used for grounding, which is Google's word for feeding pages from the search index to the model while it writes an answer, in the Gemini app and on Vertex AI. The same page promises the token "does not impact a site's inclusion in Google Search nor is it used as a ranking signal" [Claimed, primary accessed, S16]. What Google-Extended does not touch is AI Overviews or AI Mode. Those are governed by the Search Console toggle, which in turn leaves the Gemini app alone [Verified, S17].

Read those two documents together and you get the wrinkle. A site that accepts Cloudflare's new default has, without being told so, also asked the Gemini app to stop grounding on its pages, because Google's token bundles training with grounding and Cloudflare's setting is named for only one of them. And that same site is exactly where it was with AI Overviews, because nothing in Cloudflare's change touches them at all. That is our reading of two documents side by side, not something either company has said [Inferred (ours)]. Google has told Cloudflare that URL-level reporting for Google-Extended is coming "in the weeks to come" [Claimed via Cloudflare, S2].

Apple is simpler. The line is a Disallow for Applebot-Extended [Verified, S2]. Apple's support page, which carries a date of 4 September, eleven days before Cloudflare's announcement, says the token opts a site out of training Apple's foundation models, that pages which disallow it "can still be included in search results", and that its rules "are not considered in ranking for Search" [Claimed, primary accessed, S19]. What Apple's models may quote from a page is a separate control: a nosnippet tag, or marking a paywalled page as not free to access, stops Applebot using it "as additional context" when a model writes an answer [Claimed, primary accessed, S19]. A URL-level inspection tool is, in Cloudflare's words, an "in-progress solution for next year" [Claimed via Cloudflare, S2].

Microsoft is the odd one out. There is nothing to write, because Bing has no robots.txt token for training. Cloudflare says Microsoft is "currently building the mechanism to also respect a 'no training' preference in robots.txt", "targeted for early 2027" [Verified, S2]. Until then, Disallow AI Training tells Bingbot nothing whatsoever. Microsoft's control is the same NOARCHIVE meta tag it announced in September 2023, which keeps a page out of Bing Chat, now Copilot, answers and out of training at the same time, while the page "will still appear in our search results" [Claimed, primary accessed, S20].

One more change hides in the wording. Under the old switch, Block "excludes mixed-purpose bots that are used both for Training and for Search", in the words of Cloudflare's documentation [Verified, S6]; Googlebot was never touched by it. Under the new controls, Block means block. A site that wants the three stopped is told to "Select Block" [Verified, S2]. So the setting that used to be safe for Google traffic now isn't, and the one that keeps Google traffic is the one with the friendlier name. We don't tell readers which to choose. We do think the word deserves a second look.

Who still gets to read you

Training was never really our subject. Citation is. When an engine names a company in an answer, it got there through a crawler, and the question for anyone who sells anything is which crawlers the new settings still let in. Table 2 lays Cloudflare's own bot directory against the migrated settings. The categories are the directory's [Verified, S5]; the outcomes for the crawlers Cloudflare names are the post's [Verified, S2]; the step of mapping the directory's four categories onto the three new controls is ours, and could be wrong at the edges [Inferred (ours), S3, S6].

Table 2: What a migrated site now does to each crawler (Search: Allow. Training: Disallow AI Training. Agent: Block on pages with ads.)

Two rows do the work. None of the index crawlers behind ChatGPT search, Claude, Perplexity, Google's AI features or Copilot is blocked by the migrated settings. The live fetchers, the bots that go and read a page while a person waits for an answer, are a different story. On any page Cloudflare's detector decides is carrying an ad, ChatGPT-User, Claude-User and Perplexity-User are turned away. For an ad-funded publisher that is the article page, the one you would most want an engine to read. The detector parses the page for ad code, reads Content Security Policy reports and runs a filter list cut down from more than 40,000 ad-blocker patterns to about 400 [Verified, S21]. What an engine does when a live fetch is refused, whether it falls back to its index copy, drops the source or quietly names someone else, is not something any engine has disclosed, and we found nobody who has measured it [not found].

There is no Disallow option for agents. Cloudflare's reason is that "the Internet does not yet have a well-established directive for expressing Disallow preferences to agents"; it will revisit this as standards such as ai-prefs mature [Verified, S2].

What Cloudflare did not say

Perplexity is not in the announcement. Not among the accountable operators, not among the blocked ones, not anywhere [Verified: absence, S2]. Cloudflare's own directory lists PerplexityBot under AI Search and Perplexity-User under AI Assistant [Verified, S5]. Thirteen months ago Cloudflare took Perplexity off its verified-bot list in an unusually public way, publishing evidence that it had used undeclared crawlers and rotating IP addresses to get round sites that had told it to stay out [Verified, S22]. Whether Perplexity has since been let back on, and whether it is being asked to meet the four conditions, is not stated on any page we read [not found]. For a company whose search bot gets a green light in the table above, that is a large silence.

The two numbers Cloudflare uses to justify the design are thinner than they look. "Less than 1% of Cloudflare sites choose to block Search bots" and "17% of sites choose to enable some mechanism to block training" come with no base, no period and no method [Claimed, S2]. For scale, Cloudflare said in July 2025 that more than a million customers had used the block switch in its first year [Claimed, S21], and in December that it had blocked 416 billion AI-bot requests since 1 July 2025 [Claimed via press, S11].

Then there is the Pew number. The post says people who read an AI summary are "over 40% more likely to end their search after reading one", and cites Pew. We went to Pew's page. Its study, of the March 2025 browsing of 900 US adults, found that 26% of visits with an AI summary ended the session against 16% without, and that people clicked a normal result in 8% of visits with a summary against 15% without [Verified, S23]. Those are Pew's numbers. The "over 40%" is Cloudflare's, and we could not find it on Pew's page. The claim that AI-search referrals convert at "between three times and over five times" the rate of ordinary search links to a Similarweb blog post and to a vendor's round-up of statistics [Claimed, vendor, S2].

And the page that is meant to make the whole arrangement checkable, the "AI bot transparency" table on Radar, which Cloudflare says "publicly tracks the controls, transparency, and reporting provided by Accountable crawler operators" [Verified, S2], would not load for us. It needs JavaScript and our capture does not run it [not read, S26]. We will go back with a proper browser and add what it says.

India

One more thing belongs in the same file, because this publication runs its tests here. In December the DPIIT published a working paper proposing a mandatory blanket licence for AI training in India, paid for under statute and, in Business Standard's summary, with no way out: "Copyright owners cannot opt out of allowing their works to be used for AI training" [Claimed via press, S24]. Comments closed on 6 February [Verified, S25]; we found no later version [not found]. If that became law, the sign on the lawn would be sitting inside a statute that says come in, for a fee. That is our reading, not the paper's [Inferred (ours)].

The dates

A record is only useful if someone comes back to it, so here is what we will be checking. Google's URL-level reporting for Google-Extended, promised "in the weeks to come" from 15 September [Claimed via Cloudflare, S2]. Apple's URL-level tool, due "next year" [Claimed via Cloudflare, S2]. Microsoft's robots.txt support, "targeted for early 2027" [Verified, S2]. Cloudflare's own control over "how much of your content is included" in a summary, set once rather than company by company, promised "by early next year" [Verified, S2]. Google's nine months under the CMA, which run out in early March 2027 on our count [Inferred (ours), from S14]. And our own weekly panel, which starts logging cited domains from its first week, with 15 September 2026 marked in the log as the day the fence became a sign for the three companies whose crawlers Cloudflare had spent a year trying to split.

Labels used in this article

Sources

All pages captured 16 September 2026 unless stated.

S1. Cloudflare customer email, "Updates to managing AI crawlers on your account", from Bryan Becker, Director, Product Security, September 2026. Received by Salience Index.

S2. Cloudflare blog, "Have it both ways: stay discoverable in search while disallowing AI training", Bryan Becker, 15 September 2026. https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/

S3. Cloudflare blog, "Your site, your rules: new AI traffic options for all customers", Jin-Hee Lee and Bryan Becker, 1 July 2026 (modified 15 July 2026). https://blog.cloudflare.com/content-independence-day-ai-options/

S4. Cloudflare blog, "Say it once: introducing Bot Preference Sync", Jin-Hee Lee, 21 August 2026. https://blog.cloudflare.com/bot-preference-sync/

S5. Cloudflare Docs, AI Crawl Control, "Bot reference", last updated 23 April 2026. https://developers.cloudflare.com/ai-crawl-control/reference/bots

S6. Cloudflare Docs, Bots, "Block AI bots", last updated 1 July 2026. https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots

S7. Search Engine Journal, "Report That Cloudflare AI Bot Blocking Prevents Googlebot From Indexing Sites", Roger Montti, 4 August 2026. https://www.searchenginejournal.com/report-that-cloudflare-ai-bot-blocking-prevents-googlebot-from-indexing-sites/584673/

S8. Search Engine Journal, "Cloudflare's AI Crawler Rules Can Block Googlebot", Matt G. Southern, 2 July 2026. https://www.searchenginejournal.com/cloudflares-ai-crawler-rules-can-block-googlebot/581385/

S9. Playwire, "Cloudflare's AI Bot Blocker Is Also Blocking Googlebot", 5 August 2026. https://www.playwire.com/blog/cloudflares-ai-bot-blocker-is-also-blocking-googlebot

S10. Cloudflare blog, "Content Independence Day: no AI crawl without compensation!", Matthew Prince, 1 July 2025. https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/

S11. Search Engine Land, "Cloudflare: 416 billion AI bot requests blocked since July", Danny Goodwin, 5 December 2025 (reporting Matthew Prince's remarks at WIRED's Big Interview). https://searchengineland.com/cloudflare-416-billion-ai-bot-requests-blocked-since-july-465704

S12. Cloudflare, response to the CMA consultation on Google's publisher conduct requirements, 25 February 2026. https://assets.publishing.service.gov.uk/media/69b970d9f6acd3f43e96129b/Cloudflare.pdf

S13. Cloudflare blog, "Google's AI advantage: why crawler separation is the only path to a fair Internet", Maria Palmieri and Sebastian Hufnagel, 30 January 2026 (modified 22 July 2026). https://blog.cloudflare.com/uk-google-ai-crawler-policy/

S14. GOV.UK, Competition and Markets Authority, "CMA secures fairer deal for publishers and improves Google search services in UK", 3 June 2026. https://www.gov.uk/government/news/cma-secures-fairer-deal-for-publishers-and-improves-google-search-services-in-uk

S15. PPA, "CMA publishes final decision on Google Search publisher controls", 4 June 2026. https://ppa.co.uk/cma-publishes-final-decision-on-google-search-publisher-controls

S16. Google Search Central, "Google's common crawlers" (Google-Extended entry), last updated 14 July 2026. https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers

S17. Google, The Keyword, "New opportunities, control and insights for website owners", Mrinalini Loew, 3 June 2026 (updated 31 August 2026). https://blog.google/products-and-platforms/products/search/new-controls-website-owners/

S18. Search Engine Land, "Google Search Console AI performance reports and Search generative AI control rolling out globally", Barry Schwartz, 31 August 2026. https://searchengineland.com/google-search-console-ai-performance-reports-and-search-generative-ai-control-rolling-out-globally-486269

S19. Apple Support, "About Applebot" (Applebot-Extended section), page dated 4 September 2026. https://support.apple.com/en-us/119829

S20. Bing Webmaster Blog, "Announcing new options for webmasters to control usage of their content in Bing Chat", 22 September 2023. https://blogs.bing.com/webmaster/september-2023/Announcing-new-options-for-webmasters-to-control-usage-of-their-content-in-Bing-Chat

S21. Cloudflare blog, "Control content use for AI training with Cloudflare's managed robots.txt and blocking for monetized content", Jin-Hee Lee and others, 1 July 2025. https://blog.cloudflare.com/control-content-use-for-ai-training/

S22. Cloudflare blog, "Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives", 4 August 2025. https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/

S23. Pew Research Center, "Google users are less likely to click on links when an AI summary appears in the results", Athena Chapekis and Anna Lieb, 22 July 2025. https://www.pewresearch.org/short-reads/2025/07/22/google-users-are-less-likely-to-click-on-links-when-an-ai-summary-appears-in-the-results/

S24. Business Standard, "Centre recommends mandatory AI licence, royalties for copyrighted works", 9 December 2025. https://www.business-standard.com/industry/news/dpiit-proposes-mandatory-ai-copyright-licence-law-data-training-125120900662_1.html

S25. Storyboard18, "DPIIT extends consultation on generative AI–copyright framework till February 6", 5 January 2026. https://www.storyboard18.com/digital/dpiit-extends-consultation-on-generative-ai-copyright-framework-till-february-6-87006.htm

S26. Cloudflare Radar, "AI Insights" (section "AI bot transparency"), captured 16 September 2026; content not readable without JavaScript. https://radar.cloudflare.com/ai-insights