Yandex compares MCP and CLI+Skill for AI agents: 400 requests and an unexpected breakdown

The Yandex City Services team ran a benchmark comparing two ways of connecting an AI agent to internal APIs, and found that the architectural choice directly affects how efficiently tokens are spent.

The problem: tokens are finite

The context window is limited; everyone knows that. But few people count how many tokens go not to the task itself but to the scaffolding: tool descriptions, parameter lists, intermediate results of calls. In complex scenarios this overhead can take up a significant share of the available context. The agent then starts making mistakes not because the model is bad, but simply because no useful space is left.

Daniil Mikhailov from Yandex's partner products team put the question bluntly: how can you do more while spending fewer tokens when working with real internal APIs?

MCP vs. CLI + Skill

The team compared two ways of integrating an agent with tools.

MCP (Model Context Protocol) is a structured protocol: the agent receives a description of each tool in an explicit format, and calls go through a standardized layer. The upside is universality and a predictable schema. The downside is that every tool description occupies context space in full.

CLI + Skill is an alternative approach: the agent uses the command line, and knowledge about the tools is packed into a compact "skill", a prewritten instruction prompt. The description is more compact, but it has to be maintained by hand.

To test a hypothesis drawn from outside research, the team built a benchmark:

• 14 real scenarios involving Yandex's internal tools
• 2 language models
• More than 400 requests
• Accuracy and token-cost measurements in each scenario

The moment everything broke

The most valuable thing happened not at the end but in the middle of the experiment: something that had been working reliably suddenly stopped. According to Mikhailov, this failure turned out to be more interesting than the final numbers, and the team had to dig into the causes.

"At some point everything that had been working broke, and that turned out to be the most interesting part.

We had to figure out why."

Anomalies like this in benchmarks often expose hidden dependencies: how the model interprets the schema format, how tools behave on repeated calls, how stable the output is across different phrasings of the task. Without such a stress moment, the results could have been naively optimistic.

The result: a decision tree

After the series of experiments, the team built a practical decision tree: when MCP pays off, and when CLI + Skill does. This is not an abstract recommendation but a conclusion drawn from real data: more than 400 requests on real infrastructure.

What this means

Choosing how to connect an agent to an API is not a technical whim. It affects how many tokens are wasted, how long the context lasts, and how stably the agent behaves in non-standard scenarios. For teams building product agents on top of internal systems, this study offers a concrete tool for choosing an architecture, based on real measurements rather than marketing.