Cloudflare set a deadline for AI crawlers: from September 2026 — pay publishers or face blocking

Cloudflare will start blocking AI crawlers by default on all pages with advertising from September 2026: unless the site owner explicitly grants access, data collectors won't get through. The company occupies a position before a significant share of global web traffic and intends to use this leverage to stop the free distribution of content to AI companies.

How the opt-in mechanism works

The new rule flips the familiar scheme. Until now, pages were open to everyone, and the publisher had to close access themselves through robots.txt or firewall settings — separately for each crawler. Now the logic is reversed: a page with advertising is closed to AI by default, and the company that wants the data must first obtain explicit permission from the site owner.

Cloudflare stands between user browsers and servers of a huge number of sites worldwide. This is what makes the change large-scale: one rule at the network infrastructure level closes all the company's clients — from an independent blog to a major media portal — without additional effort on their part.

• Effective date — September 2026
• All pages with advertising inventory fall under the block (unless there is explicit opt-in)
• The site owner can grant access manually — for a specific crawler or for all
• Cloudflare controls a significant share of global web traffic

Why publishers couldn't cope alone

For years, AI companies collected content for model training, ignoring or technically circumventing restrictions. Robots.txt is not legally binding: OpenAI, Perplexity, and other companies have been caught violating directives repeatedly. Lawsuits are expensive and take years: NYT vs. OpenAI and Getty Images vs. Stability AI are still unresolved, and small publishers can't afford to litigate. Direct licensing agreements went only to the largest media groups that had negotiating power.

Small and medium resources — thematic portals, news blogs, niche media — remained effectively defenseless. Content went into training datasets, advertising revenue fell as search traffic was displaced by AI answers, and there was no compensation. Cloudflare offers a systematic answer: close access at the infrastructure level — centrally and immediately for all its clients.

What this means for AI companies

If laboratories don't reach agreements with enough publishers by September 2026, their crawlers will lose access to a significant portion of current web content. Training on fresh data will become more expensive, and the search for alternative sources — synthetic data, archives, licensed partnerships — will require time and resources.

Strategically, the precedent is important: if Cloudflare's "CDN as intermediary between AI and content" model takes hold, other infrastructure providers may join in. For the AI industry, this means a gradual transition from essentially free data access to a licensing system — the one the publishing market has been demanding since the beginning of the race for training corpora.

"Stop giving the web away for free," — this is how TNW formulates

Cloudflare's position.

What this means

Cloudflare is moving the dispute between the AI industry and publishers from the legal field to the level of network infrastructure. From September 2026, crawlers risk losing a significant portion of training data if they don't reach agreements and pay. For the AI market, this is not a one-time lawsuit, but a systematic mechanism that could become the new standard for the entire CDN industry.