Cloudflare to block AI bots by default — without website owners' consent

Cloudflare announced in July 2026 a change to its default policy: mixed-purpose bots — those that simultaneously index sites for search engines, act as AI agents, and are used for training language models — are now blocked automatically on all resources protected by the company without any additional action from the site owner.

What Changed in Cloudflare's Policy

Until now, Cloudflare customers could configure their own policy regarding mixed-purpose bots. The relevant toggle was available in the personal account dashboard, but most small site owners either didn't know about it or couldn't keep up with the constantly growing list of such bots.

Now the default position is changing dramatically:

• Blocking mixed-purpose bots is enabled automatically for all Cloudflare clients
• Crawlers that combine indexing for search engines with AI agent functions or training data collection fall under the restriction
• Unblocking a specific bot remains available — but only as a conscious active step by the site owner
• The company plans to regularly update the list of recognized mixed-purpose bots

These are so-called "dual-purpose" crawlers: officially they identify themselves as search agents, but in parallel their traffic goes into training datasets for language models or is used by AI agents working in real time.

Why Cloudflare Is Changing Course Now

The generative AI boom has dramatically increased server load: the volume of requests from bots has grown exponentially over the past two years. Many sites are finding that a significant portion of their traffic comes not from real users, but from crawlers that often don't identify themselves as AI tools.

At the same time, a wave of lawsuits is mounting: publishers, photographers, and developers are suing AI companies for using their content without permission to train models. Such cases are already being heard in US and EU courts.

"The default position will be defensive in nature,"

Cloudflare said in an official statement.

Cloudflare serves millions of sites worldwide and processes a significant share of global web traffic. This decision shifts the default balance of power between content creators and those who use it to train AI.

What This Means

For site owners and small publications, the change means automatic protection without additional setup — these are the ones who most often fail to respond when new bots appear. For AI product developers using crawlers to collect data or create agents with access to up-to-date web content, the situation becomes more complicated: they will have to negotiate with each resource explicitly or switch to licensed datasets. Major players — OpenAI, Google, Perplexity — are already signing such agreements with publishers, and this trend will grow as platform policies become stricter.

Frequently Asked Questions

How do I allow a specific bot access to my site?

In the Cloudflare control panel, you can add an explicit exception for the needed crawler. The default blocking is not mandatory and can be disabled in the personal account in a few clicks.

Will standard search bots be blocked?

According to the company, the restriction targets mixed-purpose bots — those that simultaneously index content and use it for AI tasks. Standard search crawlers do not fall under the new policy.