Should we allow AI crawlers at all?

It is a real decision with a real cost on both sides, and it should be made deliberately rather than inherited from a default. Blocking does not remove your brand from AI answers; it removes your version of the story. The model still answers questions about you using whatever it can read — review aggregators, complaint threads, competitors. In our audits, blocked sites consistently received the harshest verdicts.

We already allow them in robots.txt but the crawler still gets nothing. why?

Two usual causes. Either the block is not in robots.txt at all but on the CDN edge, where a managed rule overrides whatever your repository says, or the page renders client-side so the crawler receives a valid empty shell. Both look fine from inside your codebase, which is why the only reliable check is requesting the page as the crawler and reading the body.

Does blocking protect our content from being used for training?

Partly and imperfectly, and it is worth separating the two goals. Some agents are declared for training and some for answering user questions in real time; blocking the second group costs you visibility today in exchange for uncertain protection. Decide per agent rather than with one blanket rule.

robots.txt for AI crawlers

Robots rules for AI crawlers are simple to write and easy to get wrong in ways nobody on the team can see. In our 100-audit dataset half the sites were unreadable to AI crawlers to some degree and a third were a complete blackout — and in a meaningful share of those the team believed the crawlers were allowed.

the agents worth naming

who is asking, and what for
user-agent	who	what it is for
GPTBot	OpenAI	Crawling for model training
OAI-SearchBot	OpenAI	Search results inside ChatGPT
ChatGPT-User	OpenAI	Fetching a page a user asked about right now
ClaudeBot	Anthropic	Crawling
PerplexityBot	Perplexity	Indexing for answers
Google-Extended	Google	Gemini grounding, separate from Googlebot
CCBot	Common Crawl	The open corpus many models are built on

Naming them explicitly beats relying on a wildcard, because it makes the decision visible to whoever reads the file next. Ours names eleven and allows them; the file is at 009.agency/robots.txt.

the trap: the rule that is not in your repository

The most expensive pattern we find is a site whose robots.txt allows AI crawlers while a managed rule at the CDN blocks them anyway. The file in the repository says one thing, the edge says another, and the edge wins. Nobody on the engineering team can see it by reading the codebase, so it survives audits and redesigns.

We have audited sites where the team had explicitly tried to open the crawlers up, and their own change had no effect for exactly this reason. If your crawler check fails while your robots.txt looks correct, the CDN is the first place to look.

how to verify, in one command

Request the page the way the crawler does and read what comes back:

curl -A "GPTBot/1.0" https://your-site.com/

Three failure signatures. A 403 or 401 means something is blocking by user-agent. A redirect loop usually means bot protection. A 200 with almost no text means rendering, not robots — the crawler got the page and the page was empty.

Repeat for each agent you care about. Agents are treated differently by most protection layers, so one passing does not mean the rest do.

making the decision honestly

Allowing everything and blocking everything are both defensible; drifting into one by accident is not. A workable middle: allow the agents that answer user questions in real time, decide separately about training crawlers, and write the reasoning into a comment in the file so the next person inherits the decision rather than the mystery.

Whatever you choose, verify it from outside afterwards. The gap between what a config says and what a crawler receives is where most of this problem lives.