Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseInsight · On RadarCloudflare AI Blog→ original

Cloudflare Will Show Where Teams Overpay for Oversized AI Models

AI bills keep climbing, but token counts don't show where the money goes. Cloudflare has added an Overkill section to AI Gateway. It finds conversations where a simple task, such as formatting or a short summary, was sent to a heavy reasoning model. Alongside it, you can see the users and agents involved, along with cost, latency, and the number of turns. That makes it easier for teams to decide where to change their default model.

AI-processed from Cloudflare AI Blog; edited by Hamidun News
Cloudflare Will Show Where Teams Overpay for Oversized AI Models
Source: Cloudflare AI Blog. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceCloudflare AI Blog ↗
Editor-in-Chief
Key metric #1The features are free for AI Gateway users
Analysis runs asynchronouslydata lags traffic by about a day
Key metric #3Auto Router, in beta, picks a model for each task on its own and not always the cheapest one

A few weeks after a company rolls out AI, the same story almost always repeats: costs rise, some requests respond more slowly, and nobody can pinpoint the cause. Developers may have taken on more complex code. Agents may have started making unnecessary clarifying calls. Or a small group of people may be spending far more than their share. Token and request counters can't tell these cases apart. Cloudflare set out to close that gap with an update to User Insights in its AI Gateway.

What the Overkill section actually shows

User Insights appeared in AI Gateway a month earlier. It showed which users, applications, and models generated traffic, and it flagged cost anomalies. Now it adds context: what kind of work sits behind a request. Is it code review, research, or an agent that made several calls in a row?

The Overkill section looks for conversations where the chosen model seems stronger than the task requires. A typical example is text formatting or a short summary sent to an expensive reasoning model. A everyday analogy fits: moving one carton of milk in a truck. It works, but it's expensive.

Cloudflare stresses that this is not a model ranking and not an automatic recommendation to switch. The section helps teams ask questions. Does the model suit the task? Does the extra capability improve the result? Could a fast, cheap model handle it? Is the problem confined to one process, one user, or one agent? The causes vary: a model is set as the default, people don't know which one to choose, or an agent is configured to use the same model for every step.

Alongside it, the Potential Savings view works. It shows requests that could potentially move to a faster or cheaper model without losing quality. You can also compare latency, input and output tokens, turn counts, and total cost for a single task type.

How the service understands the task type

The analysis groups conversations into categories: programming, research, writing, summarization, and data analysis. Turn count is analyzed separately. A long conversation isn't necessarily bad, especially in complex work. But if a simple task keeps needing several passes, it's worth checking the prompt, the model, or the process. Moreover, the cost of a task isn't determined by the first request alone. It's the sum of time, tokens, and money spent until the final result.

Technically, a separate Cloudflare Worker performs the classification. It reads the conversation trajectory: user requests, assistant replies, tool calls, and their results. The output is a category, a confidence score, and several parameters: task complexity, intent ambiguity, cost of error, and context dependence.

An important safety note: metadata is stored separately from log bodies (Durable Objects for metadata, R2 for bodies). The dashboard shows derived categories and aggregates, not the prompts themselves. The retention period for raw logs is determined by current AI Gateway settings, so check them in advance.

Another limitation: classification is asynchronous. It runs after a request is processed and adds no latency for the user. But dashboard data may lag by roughly a day. This is a tool for analyzing trends, not for monitoring traffic in real time.

Who ends up in the report, and where Auto Router fits

To link spending to people, AI Gateway uses identification. For your own applications, requests need stable user_id and session_id values, plus optionally a group and an application name in the cf-aig-metadata header. Cloudflare advises passing non-personal identifiers and not putting data about people into the prompt. For tools like Claude Code, Codex, and OpenCode, it's enough to place AI Gateway behind Cloudflare Access: user and session context is pulled in automatically. Access is free for teams of up to 50 people.

The second part of the release is Auto Router. It takes the same signals (conversation trajectory, category, complexity, and how well the model fits the task) and selects a model with cost in mind. The idea is to avoid writing a separate routing rule for each scenario. Complex code or research can still go to a powerful model, while simple tasks go to a fast, cheap one. The announcement describes Auto Router as being in both public and closed beta; sign-up runs through a separate Cloudflare post, so check access separately.

What to do in practice

If your agents and staff all use one top-tier model, start by checking what share of traffic goes to summaries, formatting, and classification, and how many turns those tasks take. If most conversations are resolved in a single turn, that's a clear candidate for a cheaper model. Cloudflare doesn't publish savings figures, so you'll need to test the hypothesis on your own data. But the approach itself is sound: understand the work first, then choose the model. All the described User Insights features are free for AI Gateway users.

Story

Key Questions Answered

What the Overkill section actually shows?

User Insights appeared in AI Gateway a month earlier. It showed which users, applications, and models generated traffic, and it flagged cost anomalies. Now it adds context: what kind of work sits behind a request. Is it code review, research, or an agent that made several calls in a row? The Overkill section looks for conversations where the chosen model seems stronger than the task requires. A typical example is text formatting or a short summary sent to an expensive reasoning model.

How the service understands the task type?

The analysis groups conversations into categories: programming, research, writing, summarization, and data analysis. Turn count is analyzed separately. A long conversation isn't necessarily bad, especially in complex work. But if a simple task keeps needing several passes, it's worth checking the prompt, the model, or the process. Moreover, the cost of a task isn't determined by the first request alone. It's the sum of time, tokens, and money spent until the final result.

Who ends up in the report, and where Auto Router fits?

To link spending to people, AI Gateway uses identification. For your own applications, requests need stable user_id and session_id values, plus optionally a group and an application name in the cf-aig-metadata header. Cloudflare advises passing non-personal identifiers and not putting data about people into the prompt. For tools like Claude Code, Codex, and OpenCode, it's enough to place AI Gateway behind Cloudflare Access: user and session context is pulled in automatically.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Private AI Builders Community · Vibe Club

Ship AI Faster with Founders, Engineers & Weekly Architecture Reviews

Inside Vibe Club (hamidun.com/club) — weekly live masterminds with Zhemal Khamidun, Tech Radar, full LMS access, and 370+ production Claude Code skills.

Join Vibe Club →Explore Hamidun Academy
What do you think?
Loading comments…