New rules for AI companies: Cloudflare will require split crawlers by September 15
Cloudflare has given AI companies until September 15, 2026 to split their web crawlers: search bots must be separate from bots used to train AI models and agents. Companies that fail to comply will be blocked by default on publishers' sites. The aim of the policy is to push AI companies to pay for content rather than take it for free.
AI-processed from TechCrunch; edited by Hamidun News
On July 1, 2026, Cloudflare introduced new requirements for AI companies: they must split their web crawlers, so that bots for search and bots for training AI models are separate entities, by September 15, 2026.
Why is Cloudflare separating AI companies' crawlers?
The aim is to push AI companies to pay for content instead of using publishers' data without compensation.
What happens to AI companies that do not comply?
Cloudflare's infrastructure will start blocking them by default on publishers' sites.
Key Questions Answered
Why is Cloudflare separating AI companies' crawlers?
The aim is to push AI companies to pay for content instead of using publishers' data without compensation.
What happens to AI companies that do not comply?
Cloudflare's infrastructure will start blocking them by default on publishers' sites.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.