Cloudflare will block AI bots by default, without site owners opting in

In July 2026, Cloudflare announced a change to its default policy: mixed-purpose bots, meaning those that index sites for search engines, act as AI agents and are used to train language models, are now blocked automatically on all resources the company protects, with no action required from the site owner.

What changed in Cloudflare's policy

Until now, Cloudflare customers could set their own policy toward mixed-purpose bots. The relevant switch was available in the dashboard, but most owners of small sites either did not know about it or could not keep up with the constantly growing list of such bots.

The default position is now changing radically:

• Blocking of mixed-purpose bots is enabled automatically for all Cloudflare customers
• The restriction covers crawlers that combine indexing for search engines with AI-agent functions or the collection of training data
• Unblocking a specific bot is still available, but only as a deliberate, active step by the site owner
• The company plans to update the list of recognized mixed-purpose bots regularly

These are so-called "dual-purpose" crawlers: officially they present themselves as search agents, but at the same time their traffic flows into language-model training datasets or is used by AI agents working in real time.

Why Cloudflare is changing course now

The generative AI boom has sharply increased the load on servers: the number of requests from bots has grown several-fold over the past two years. Many sites report that a significant share of their traffic is not real users but crawlers, which often do not identify themselves as AI tools at all.

At the same time, a wave of lawsuits is growing: publishers, photographers and developers are suing AI companies that use their content without permission to train models. Such cases are already being heard in courts in the US and the EU.

"The default position will be defensive," says

Cloudflare's official statement.

Cloudflare serves millions of sites worldwide and handles a significant share of global web traffic. The decision shifts the default balance of power between content creators and those who use their content to train AI.

What this means

For resource owners and small publications, the change means automatic protection without extra configuration; they were the ones who most often failed to react to new bots appearing. For developers of AI products that use crawlers to collect data or to build agents with access to current web content, the situation gets harder: they will have to negotiate with each resource explicitly or switch to licensed datasets. Major players such as OpenAI, Google and Perplexity are already striking such agreements with publishers, and the trend will grow as platform policies tighten.

Frequently asked questions

How do I allow a specific bot to access my site?

In the Cloudflare dashboard you can add an explicit exception for the crawler you need. The default block is not mandatory and can be lifted in a few clicks through your account.

Will ordinary search bots be blocked?

According to the company, the restriction targets mixed-purpose bots, those that index content and also use it for AI tasks. Standard search crawlers are not covered by the new policy.