Cloudflare: AI agent requests up 1,700%, and sites are being told to charge them
Search engines used to send people to websites, and people brought ads and subscriptions. Now, according to Cloudflare, AI agent requests grew more than 1,700% in a year, and for the first time, more than half of all traffic is created by non-humans. An agent reads a page and summarizes it for the user, so the site spends resources without getting a visitor. The conclusion: separate bots by purpose and charge agents, rather than blocking everyone indiscriminately.
AI-processed from Cloudflare AI Blog; edited by Hamidun News
Picture a small review site. People used to read it after arriving from search, watching ads, and occasionally buying a subscription. Now people ask a chatbot, and an AI agent visits the site on its own, pulls the relevant information from the page, and summarizes it for the user.
The site pays for bandwidth and server capacity, but there is no visitor and no ad impression. Cloudflare describes this exact problem in its post "The internet has a second audience." According to the company, daily requests from AI agents (programs that act on a person's behalf) grew more than 1,700% over the year.
This year, for the first time, more than half of all internet traffic was generated by non-humans.
What is happening to traffic
At the end of 2024, the Cloudflare network handled an average of 63 million HTTP requests per second. Now it is around 115 million, with peaks above 150 million. Humans have not gone anywhere, but in the most heavily crawled categories (retail, software, IT services, and finance), human traffic fell by as much as 40% in less than a year.
The share of crawlers that collect text to train models is also growing: in spring 2025 they accounted for 22% of crawler requests, and by June 2026 that figure reached 52% (according to the bots' own stated purpose).
Cloudflare distinguishes two types of bots. A training crawler collects pages once. An agent comes back every time someone asks about that content, so its traffic grows with the number of questions, not with the volume of what you publish. Some agents also act as shoppers: they book a table or compare insurance policies. Refusing such an agent may mean refusing the person who sent it.
How they propose to earn money
The first step is to see who is arriving. Since July, the single "block AI bots" toggle has been replaced with three separate controls: for search, for agents, and for training. They are available on all plans, including the free one.
The statistics explain why this matters: fewer than 1% of sites block search crawlers, while 17% block training crawlers. Since September 15, there is also a Disallow AI Training option: the site stays in search results, but the bot operator is informed that the data may not be used for training. Apple, Google, and Microsoft have pledged to respect it.
To tell genuine agents from impostors, Web Bot Auth is used: OpenAI, Google, and AWS cryptographically sign requests from their agents.
The second step is money, and here there are two beta products.
Pay Per Use pays the publisher not for crawling a page, but for the actual use of content. The buyer is a verified crawler, which itself defines what counts as use and how much it pays for it. The publisher sees the offer and may accept or withdraw at any time, while Cloudflare checks the buyer's reports, issues invoices, and transfers the money. The scheme is designed for those who cannot expect a personal deal with an AI company, for example a trade journal for marine engineers. Since 2023, more than 50 licensing deals have been signed between publishers and AI companies, almost all of them individual deals between major players.
Monetization Gateway works like a checkout counter at the entrance. You set prices for requests through rules, and when a rule triggers, the agent receives an HTTP 402 Payment Required response and pays the seller directly over the open x402 protocol. The price can be per request, per database query, or per token, either fixed or with a cap. The post gives an example: a sports site charges a fraction of a cent when an agent asks "who leads in assists." This is a closed beta for eligible customers in the US. Thousands of sellers have joined the waitlist, and the most common request is "charge the agents, not the people."
Costs and caveats
There is also another side: many crawlers repeatedly download pages built for humans just to extract a couple of paragraphs of text, and they visit sites that have not changed. To address this, Cloudflare offers Markdown for Agents, which lets an agent read a page without its markup, and WebMCP, which lets a site expose the actions it supports instead of forcing the agent to guess which button to press. The dashboard already shows how much traffic each operator consumes. Results of a joint study with OpenAI on efficient indexing are promised in the coming weeks.
The caveats are obvious. Cloudflare wrote the post itself, and it has a commercial interest: its network sits in front of more than 20% of the web. The authors themselves call pricing and content discovery methods unresolved, which is why both products are in beta. Whether the scheme works depends on whether AI companies are willing to pay.
The practical takeaway: if you run a site, configure bot policy by purpose rather than with a single switch. If you have an API or MCP tools, plan for per-call pricing and a 402 response. If you build agents, be aware that free access to websites may become paid.
- Nov 21, 2026A viral GTA 6 Claude clip looks fake: half an hour of control burned through the 5-hour Pro limit
- Nov 21, 2026Claude Code Speeds Up Snowboard Kids 2 Decompilation: From 20% to 45% of Functions in Three Months
- Nov 21, 2026VibeCBS collects project nicknames in advance: the agent finds 2.8 times more documents
- Nov 21, 2026FDE job postings grew 4.6x in a year: what a forward deployed engineer actually does at the client's site
Key Questions Answered
What is happening to traffic?
At the end of 2024, the Cloudflare network handled an average of 63 million HTTP requests per second. Now it is around 115 million, with peaks above 150 million. Humans have not gone anywhere, but in the most heavily crawled categories (retail, software, IT services, and finance), human traffic fell by as much as 40% in less than a year.
How they propose to earn money?
The first step is to see who is arriving. Since July, the single "block AI bots" toggle has been replaced with three separate controls: for search, for agents, and for training. They are available on all plans, including the free one. The statistics explain why this matters: fewer than 1% of sites block search crawlers, while 17% block training crawlers.
What are the key details on Costs and caveats?
There is also another side: many crawlers repeatedly download pages built for humans just to extract a couple of paragraphs of text, and they visit sites that have not changed. To address this, Cloudflare offers Markdown for Agents, which lets an agent read a page without its markup, and WebMCP, which lets a site expose the actions it supports instead of forcing the agent to guess which button to press. The dashboard already shows how much traffic each operator consumes.
Ship AI Faster with Founders, Engineers & Weekly Architecture Reviews
Inside Vibe Club (hamidun.com/club) — weekly live masterminds with Zhemal Khamidun, Tech Radar, full LMS access, and 370+ production Claude Code skills.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.