Anthropic's three bots can raise a site's chances of appearing in Claude's sources to 92%

A site can be useful, well written, and accessible to Google—yet Claude may still fail to see it during a search. A common reason is straightforward: the owner wanted to prevent the content from being used to train models and blocked all Anthropic bots with a single rule. As a result, along with the training crawler, the owner also cut off search indexing and direct-link page retrieval.

Anthropic uses different bots for these tasks. ClaudeBot collects material for model training. Claude-SearchBot indexes pages that may appear in Claude's web search results. Claude-User retrieves a specific page when a user provides a link or asks for its contents. Blocking ClaudeBot alone therefore should not deprive a site of visibility—provided the other two bots are actually allowed.

One blocking rule solves the wrong problem

Consider the documentation for a SaaS service. The team does not want it included in future training datasets, but it does want Claude to cite the instructions when customers ask questions. In that case, robots.txt can block ClaudeBot while separately allowing Claude-SearchBot and Claude-User. In simplified form, the logic is: Disallow: / for the first bot and Allow: / for the other two.

But robots.txt alone is not enough. A WAF or CDN may reject requests before the bot reads the rules. For example, a preconfigured rule for blocking AI crawlers may combine several user agents into one group. The file then permits access while the site's security layer returns a denial. The entire chain must be checked: robots.txt for every subdomain, CDN and WAF rules, and then server logs. The bot name in the user-agent should be checked against Anthropic's published IP list because that string is easy to spoof.

Indexing does not automatically mean citation

Once bots are allowed, the page still needs to enter the search index used by Claude. Anthropic does not officially identify its search provider, but several independent observations point to a strong connection with Brave Search. In one comparison, 87.8% of the pages found by Claude ranked in Brave's top 10; the overlap with Google was 45.6%. In another measurement, sites ranking 1–3 in Brave were cited in 92% of the answers studied, while pages outside the top 20 were cited in 12%.

This is correlation, not proof of a direct causal relationship. The findings also cannot automatically be generalized to Russian-language queries. Moreover, Claude does not always use web search: in the State of AEO 2026 study, search was activated in 36.6% of cases. If the model answers from memory, a newly published page cannot become a source regardless of the site's configuration.

What to check in practice

Start with robots.txt on the main domain and every subdomain. Make sure Claude-SearchBot and Claude-User are not covered by a blanket Disallow: /. Then inspect the WAF, CDN, and access logs. After that, look for the page in Brave using several queries for which it genuinely provides the best answer.

The final check is to give Claude a direct link and ask it to summarize the page, then ask a topical question using the phrase Search the web. Repeat the experiment with several phrasings: a single appearance or absence of a link proves nothing.

The practical conclusion is simple: treat opting out of training separately from opting out of search visibility. Block ClaudeBot if company policy requires it, but deliberately preserve access for Claude-SearchBot and Claude-User. This does not guarantee placement in an answer, but it removes the technical obstacle that could prevent Claude from considering your content at all.