robots.txt Doesn't Block AI Training: Which Bots to Block to Stay in AI Answers

Many site owners add a Disallow line for GPTBot to robots.txt and are sure they have closed their texts off from ChatGPT. Gary Illyes of Google reminded everyone: that is not the case. The file answers a single question: can a crawler download a page. What a system does with the content afterwards, it does not determine.

What Disallow Actually Does

The RFC 9309 standard describes robots.txt as a set of access preferences. It is a "please do not enter" sign: a well-behaved crawler will not go through the door, but the sign says nothing about what has already been seen. So a new line affects only future crawls. Pages downloaded earlier remain in the training data.

There is no general "forbid all AI" directive in the standard. Google understands only User-Agent, Allow, Disallow and Sitemap, and ignores made-up lines. A specific crawler can be blocked only by naming its token.

Three Bots Instead of One

OpenAI and Anthropic split their bots by task:

• training: GPTBot and ClaudeBot;
• the service's search: OAI-SearchBot and Claude-SearchBot;
• user requests: ChatGPT-User and Claude-User.

The settings are independent. You can block GPTBot and still stay in ChatGPT search through OAI-SearchBot. However, OpenAI states that robots.txt rules may not apply to ChatGPT-User, while Anthropic warns: disable Claude-User, and Claude will not be able to open your page at a user's request, so visibility will drop.

At Google, training is separated by the Google-Extended token. It affects Gemini training and grounding (feeding a page to the model while it answers), but not regular Search. So blocking it may cost you a place in Gemini answers. Applebot-Extended and CCBot work similarly: the former decides whether Apple's models may be trained on Applebot data, the latter blocks access for Common Crawl.

In Russia the choice is poorer. To avoid appearing in quick answers from Alice AI, Yandex advises blocking YandexAdditional and YandexAdditionalBot; changes are picked up within 2–14 days. There is no token for training alone, and a blocked YandexBot removes the site from search as well. Sber does not publish the name of the GigaChat crawler, so it cannot be named in the file.

A typical "do not train, but show" file looks like this: in the first group, User-agent for GPTBot, ClaudeBot, CCBot and Applebot-Extended with Disallow: /. Search and user tokens are allowed separately via Allow: /.

Who Ignores the Rules and What to Do

Not everyone obeys. In August 2025, Cloudflare showed that when blocked, Perplexity changes its user agent and ASN (network number), and sometimes does not even request robots.txt. The crawler presented itself as Chrome on macOS, and this went on across tens of thousands of domains. The most obedient in the same test was ChatGPT-User.

You can check your bots in the log: grep -E "GPTBot|ClaudeBot|CCBot|OAI-SearchBot|ChatGPT-User" access.log. Look at whether the crawler requested /robots.txt and whether it visited blocked sections after Disallow appeared. Token spoofing cannot be found this way, so look for requests with a browser user-agent that hit the same pages in the same rhythm. Against such bots you need protection from automated requests on the server or in a CDN, for example in Cloudflare. Blocking by IP can backfire with Anthropic: the crawler will not read robots.txt, and the refusal will not work.

There are other levers too: the nosnippet meta tag at Apple, the TDMRep protocol with the tdm-reservation=1 parameter, which carries legal weight in the EU, and Cloudflare's Pay Per Crawl, in closed beta.

What's Next

From October 30, 2026, Anthropic recognizes a general signal: User-agent __usage_ai-training with Disallow: /. Google plans to support the same AIPREF in early 2027 and, according to Lab Leaders' reading, may equate it with Google-Extended, which again puts Gemini answers at risk. The IETF standard with the Content-Usage rule remains a draft for now.

The practical takeaway is simple. First decide what you are closing off from: training or display in answers. Block training bots by name, keep search and user bots open, and verify the result in your server logs.