Cloudflare blocked AI crawlers for a year — now it is helping OpenAI index the web

Cloudflare on July 8, 2026 announced the launch of a research pilot with OpenAI: the company will test whether its network data can improve the accuracy of AI search. A notable shift — throughout the previous year, Cloudflare actively helped websites protect themselves from exactly such systems.

A Year Against Crawlers — and a Sharp Turnaround

Over the past twelve months, Cloudflare's position was unambiguous: the company developed and provided website owners with tools to block AI crawlers — automated programs that AI labs use to collect training data and index content.

This position resonated widely: publishers, bloggers, and website developers complained about aggressive scanning by OpenAI, Google, Anthropic and other AI companies, which created additional server load and occurred without explicit consent from content creators.

Now Cloudflare is changing its role — at least in experimental format. As part of the new pilot, the company will test how its network data will help OpenAI scan and index the web more accurately. Website protection tools are not going anywhere: this is about research collaboration, not a change in business model.

Why Cloudflare's Network Data Is Valuable for AI Search

Cloudflare serves traffic from more than a fifth of the entire internet. This is an exceptional position: the company sees the web not from the outside, like an ordinary crawler going through pages one by one, but from the inside — through a stream of real requests to hundreds of millions of websites and services around the world.

Such data can give an AI search engine information unavailable through traditional crawling: which resources are truly active and frequently visited, how stable a particular website operates, what content is being requested in real time. This is potentially a more accurate and up-to-date picture of the web than that collected by traditional search robots.

• Cloudflare infrastructure coverage: over 20% of global web traffic
• Pilot objective: test whether network data improves the accuracy of OpenAI's AI search
• Partnership format: research pilot, not a commercial product
• Announcement date: July 8, 2026

Context: Tension Between AI and the Open Web

The question of how AI companies collect data from the open web has become one of the industry's major conflicts over the past two years. AI system crawlers strain servers, consume traffic without monetization for publishers, and raise copyright questions. Some publishers put content behind paywalls specifically to protect against AI scanning.

Cloudflare found itself at the center of this debate: as an infrastructure provider, the company simultaneously serves AI companies and millions of websites protecting themselves from those companies. The crawler-blocking tools that Cloudflare launched last year were an attempt to take the publishers' side — and earned it a reputation as a neutral defender of the open web.

The OpenAI pilot shows that a simple choice of "for" or "against" does not exist. Cloudflare is seeking a more nuanced position: not to block AI systems indiscriminately, but to participate in shaping how these systems work with the web.

What This Means

If the research pilot demonstrates that Cloudflare's data improves AI search, this could change the architecture of the next generation of search systems: not mass page crawling by crawlers, but partnership with infrastructure providers. For Cloudflare, this opens potentially new monetization opportunities — and a new role in the AI ecosystem.