Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseInsight · On RadarHabr AI→ original

Agent instead of chatbot: AutoGen solved the task in 8,170 tokens, LangChain in 10,070

RAG is a chatbot with a knowledge-base search: it answers in a single pass, while an agent runs in a loop of "think, call a tool, check the result." In an AIMultiple test that parsed four JSON columns, AutoGen called its tools in parallel and used 8,170 tokens (units of text that providers charge for) and 47 seconds. LangChain ran its calls one after another: 10,070 tokens and 86 seconds. That is why a framework should be chosen by measuring on your own tasks, not by its popularity.

AI-processed from Habr AI; edited by Hamidun News
Agent instead of chatbot: AutoGen solved the task in 8,170 tokens, LangChain in 10,070
Source: Habr AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
Key metric #1LATS with GPT-4 solves 92.7% of HumanEval tasks on the first attempt; ReAct gets about 80%
Key metric #2In the same test, LangGraph used 9,150 tokens and 70 seconds, with 6 to 16 tool calls
Key metric #3Agents run in a sandbox with no network access by default, and tests act as the judge

Today the word "agent" gets attached to almost any chatbot, but the difference between them directly affects the token bill (tokens are the pieces of text the model provider charges for) and the risks. Ordinary RAG is a chatbot that searches your knowledge base for the relevant documents before answering: it receives a question, pulls up text, answers, and stops. An agent works differently: it calls tools itself and repeats steps until it resolves the task.

In a review on Habr, independent expert Andrey Biryukov compared the approaches and the frameworks AutoGen, LangGraph, LangChain, and CrewAI. On the same task, AutoGen used 8,170 tokens, while LangChain used 10,070. Below is an analysis focused on the engineering side and the money.

One pass versus a loop

A classic RAG system handles a query in one pass: it assembles context, generates an answer, and finishes. An agent keeps the goal and the history of its actions in mind, decides what to do next, calls a tool, looks at the result, and goes back to reasoning. The loop keeps running until the task is solved or the money and the iteration limit run out.

The article's example is "notify customers about a delay." A chatbot will write the text of an email. An agent will itself request overdue orders from the CRM, draft emails with specific dates, send them through an email API, update the status in the database, and finally check that all orders were processed.

The reasoning approaches differ. Chain-of-Thought shows intermediate steps, but errors accumulate over a long chain. Tree-of-Thought expands branches but is noticeably more expensive in computation.

ReAct alternates thought and action and remains the most practical. Reflexion writes failures into memory and takes them into account in the next attempt. The strongest of the listed variants is LATS: it applies Monte Carlo tree search, and on HumanEval (code-writing tasks) with GPT-4 it solves 92.

7% of tasks on the first attempt, while ReAct on the same model gives about 80%. The reason is that the candidate generator and the evaluator are separated, and on coding tasks the judges are tests, a linter, and type checking. An agent cannot talk itself into thinking everything is fine when the tests are red.

What the frameworks cost

The most useful part for practitioners is the AIMultiple measurement on processing unstructured data. The task: parse four independent JSON columns.

  • AutoGen: one model turn with four parallel tool calls, 8,170 tokens, median latency of 47 seconds.
  • LangGraph: 9,150 tokens and 70 seconds. The number of tool calls ranged from 6 to 16, because the system tends to repeat steps it has already completed.
  • LangChain with a sequential AgentExecutor: 4 turns, 10,070 tokens, 86 seconds.

The conclusion is simple: parallel tool calls save both time and tokens, while redundant repetition, as in LangGraph, hits the budget.

The author converts this into money at GPT-4o prices ($2.50 per million input tokens and $10 per million output tokens) for 1,000 tasks per day. That gives roughly $12–18 per day for an optimized LangGraph, $18–28 for CrewAI, and $25–40 for AutoGen.

At 10,000 tasks per day, by his estimate, the difference between frameworks reaches $47,000–80,000 per year. There is an inconsistency here: in the token test AutoGen is the most economical, but in the monetary estimate it is the most expensive. Most likely, in real scenarios the number of steps and the size of prompts differ greatly from the lab task.

The source does not disclose its calculation method, so you should run the numbers on your own data.

Memory and security

Long tasks need memory. LangGraph saves state through checkpoints, for example via AgentCoreMemorySaver, and the agent can resume after a failure. Microsoft Agent Framework creates checkpoints automatically on every workflow run. Reflexion stores short text "lessons" about failures instead of full state: this is more compact but less predictable.

Expanding an agent's capabilities also expands its attack surface. The author advises running agents in a sandbox, blocking all inbound and outbound traffic by default, and opening only the necessary connections through access lists. It is also useful to separate the agent's environment, the inference infrastructure, and the sandbox for commands, and to restrict access to credentials. The built-in code-execution settings in AutoGen and CrewAI do not replace process isolation: that remains the developer's responsibility. The second line of defense is verifying actions with external tests, and where formal verification is not possible, LLM-as-judge (another model acting as a judge) with a calibrated scale.

What to take from this

The text is mostly a survey, without the author's own experiments. Even so, three practical takeaways can be drawn from it. First: before choosing a framework, measure tokens, the number of calls, and latency on your own task. Second: wherever possible, make tool calls parallel. Third: put an objective judge over the agent (tests, a linter, validators) and keep it in an isolated environment with strict limits on iterations and budget.

Story

Key Questions Answered

What are the key details on One pass versus a loop?

A classic RAG system handles a query in one pass: it assembles context, generates an answer, and finishes. An agent keeps the goal and the history of its actions in mind, decides what to do next, calls a tool, looks at the result, and goes back to reasoning. The loop keeps running until the task is solved or the money and the iteration limit run out. The article's example is "notify customers about a delay." A chatbot will write the text of an email.

What the frameworks cost?

The most useful part for practitioners is the AIMultiple measurement on processing unstructured data. The task: parse four independent JSON columns. AutoGen: one model turn with four parallel tool calls, 8,170 tokens, median latency of 47 seconds. LangGraph: 9,150 tokens and 70 seconds. The number of tool calls ranged from 6 to 16, because the system tends to repeat steps it has already completed. LangChain with a sequential AgentExecutor: 4 turns, 10,070 tokens, 86 seconds.

What are the key details on Memory and security?

Long tasks need memory. LangGraph saves state through checkpoints, for example via AgentCoreMemorySaver, and the agent can resume after a failure. Microsoft Agent Framework creates checkpoints automatically on every workflow run. Reflexion stores short text "lessons" about failures instead of full state: this is more compact but less predictable. Expanding an agent's capabilities also expands its attack surface.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…