Contents
robots.txt has been a plain text file that you edit by hand since 1994. It is an honor system with no enforcement, which was fine when the crawlers reading it were a handful of search engines that wanted to stay on your good side. That assumption has been quietly falling apart for two years, and this week Cloudflare shipped the most interesting response to it yet.
On August 21 Cloudflare announced Bot Preference Sync, generally available on every plan from Free to Enterprise. It generates and maintains your robots.txt from the AI bot policy you already set in the dashboard. The file stops being something you write and becomes something your edge configuration emits.
The problem is drift, not syntax
The failure mode Cloudflare is targeting is specific. Your robots.txt says a crawler is disallowed. Your actual WAF and bot rules do not block it. Those two layers were configured at different times by different people, and they no longer agree.
That gap is not harmless. Cloudflare’s argument is that when your stated preference and your enforced rule disagree, some crawlers read it as evidence your preferences are stale boilerplate, and treat that as a basis to disregard them or to probe around the enforcement. The honor system degrades exactly where you were least likely to notice.
Generating the file from the enforcement config collapses the two layers into one. What you say and what you actually block become the same statement by construction.
Three categories, not one AI bucket
The policy model underneath is the part that will outlive this particular feature. Since July, Cloudflare splits AI crawler traffic into Search, Agent, and Training, and you set each independently. Search and Agent each take Allow, block only on pages that serve ads, or block everywhere. Training takes a Disallow that writes a no-training preference into the file.
That split matters because those three uses have different economics for different sites. An e-commerce store probably wants all three: it wants a shopper asking a chatbot for “the best sofa for a small apartment” to land on its product page. An ad-supported publisher wants the opposite, staying in the index that sends human readers to the page while keeping the articles out of model training. Cloudflare gives those publishers their own onboarding default, where selecting “I monetize from pages with ads” sets Training to Disallow out of the box.
The generated block is prepended between # BEGIN Cloudflare Bot Preference Sync and # END markers, and any Disallow lines you wrote yourself are preserved below it. The user agent list is refreshed from Cloudflare’s BotBase tracking, which solves the real maintenance tax: nobody hand-updates their file the week a new crawler shows up.
Transparency becomes the price of admission
The genuinely new lever is what Cloudflare asks of mixed-use crawlers, the ones that blend search, agent, and training behind a single user agent. If such a bot wants to keep crawling for search on a site that has disallowed training, its operator has to meet four conditions: honor a no-training preference by some mechanism, offer site owners a way to opt out of AI summaries, expose URL level visibility into which pages were made available for training alongside search metrics, and demonstrate publicly that disallowing training does not quietly damage traditional search ranking. Compliance is tracked on Cloudflare Radar. Crawlers that decline get blocked when you disallow training, full stop.
Read that as an accreditation regime bolted onto a voluntary protocol. Cloudflare cannot make robots.txt binding, but it sits in front of enough of the web that it can make ignoring it expensive.
Build the file
Set a policy below to see the block you would get. The user agents are representative examples of each category rather than Cloudflare’s exact BotBase list, which changes as crawlers are classified.
What I am doing about it
This site ships a static robots.txt that says Allow: / to everyone, because I want the writing read, by people and by whatever assistant they asked. I am keeping it. But it is worth being honest that it was never really a choice, it was the default I inherited from a file template.
The shift worth internalizing is that robots.txt is becoming generated output rather than a hand-edited artifact, and the input to that generator is a policy you have to actually articulate. Three questions per site: do you want to be found, do you want agents acting on your pages, do you want to be trained on. If you run on Cloudflare, spend ten minutes this week answering those deliberately instead of by default. If you do not, the categories are still the right ones to think in, and you can write the same block yourself.
Sources: Cloudflare’s Bot Preference Sync announcement and the public bots directory.