Multimodal agents lose up to 53% of their success rate when modules are assembled into a system
A recent selection of studies reveals an uncomfortable gap: strong models and accurate individual modules still do not add up to a reliable system. The SkinAgent agent identified the type of request with 80% accuracy, but completed only 47.08% of complex tasks without violations. Results in robotics look more encouraging, although many tests remain limited to simulators, synthetic speech, and isolated datasets.
AI-processed from Habr AI; edited by Hamidun News
A nearly flawless classifier does not yet make an agent reliable. That is perhaps the main practical conclusion from a recent selection of studies on multimodal models and robots. In one study, an individual module identified the required request type with 99.84% accuracy, but the assembled system fully complied with requirements in only 47.08% of complex scenarios.
The problem will be familiar to anyone who has built a product from several AI components. Recognition worked, search found the document, and the model selected a tool—yet the user still received the wrong result. The error does not necessarily occur inside a single module. It emerges at the interfaces: the agent misunderstood the intent, called the wrong tool, skipped a check, or performed the right steps in the wrong order.
Accurate components do not equal a reliable system
A good example is SkinAgent AI, a skincare assistant that analyzes a photograph, clarifies the request, and recommends products. Its authors deliberately limited it to nonmedical recommendations: the system must not make diagnoses and must follow safety rules.
The individual components looked convincing. The topical-request classifier achieved 99.84% accuracy. But on an internal set of 240 complex tasks, the agent correctly identified the user's intent in 80% of cases, selected the exact set of tools in 62.92%, and completed the entire workflow without violating requirements in only 47.08% of cases.
Imagine a support service where the operator almost always identifies the topic correctly but then, in half of all cases, opens the wrong form, forgets to verify the data, or closes the ticket prematurely. The high accuracy of the first step offers little comfort.
For a product team, the conclusion is straightforward: it is not enough to measure the quality of recognition, retrieval, or generation. An end-to-end metric is needed—the share of tasks completed correctly, safely, and without skipped steps. Tests should also include ambiguous requests, tool failures, and data that did not appear in developer demonstrations.
Internal metrics can also be misleading
Another study tested a common assumption: if the internal representations of text and an image are similar, the model must actually be using the image. The researchers replaced the image with random noise. Answer quality dropped sharply, but some alignment metrics continued to show high similarity.
In other words, the instrument reported normal conditions even after the useful signal had disappeared. For developers, this is a warning: attractive charts from hidden layers cannot be treated as proof that a model understands an image. Such indicators should be checked alongside performance on a real task. A useful minimum test is to remove the image, replace it with noise, or provide a contradictory image and see whether the answer changes as expected.
A simulator is a testbed, not an acceptance test
In robotics studies, coding agents wrote their own motion-planning programs and achieved success rates of 56–95% in several environments, compared with 47% for a conventional planner. This is a significant signal: an agent can experiment, clarify the rules of an environment, and transfer the program it discovers to new object layouts.
But the tests were conducted primarily in simulation, where the agent could see the exact state of objects. A real camera makes depth errors, has its view obstructed by objects, and changes its coordinate estimates under different lighting. The results therefore cannot be transferred directly to a warehouse or production line.
The same caution is required when evaluating voice-based fact-checking. One system achieved 86.1% accuracy, but it was tested on speech synthesized with a single voice and without real background noise. For podcasts, interviews, and live broadcasts, it remains a prototype rather than a production-ready quality guarantee.
The practical conclusion is the same: build evaluations around the complete user workflow. Test the connections between modules, deliberately corrupt inputs, and label simulation results separately. Today, much of the reliability of AI systems is lost precisely in the gap between an impressive demonstration and a working product.
- Nov 12, 2026Claude in Chrome requires approval for every click on category 3 sites
- Nov 12, 2026Norwest raises record $4 billion fund, betting on agents, voice and physical AI
- Nov 12, 2026AWS shows how 100% retrieval precision can hide an AI agent's failure
- Nov 12, 2026Garage Eight dropped peer approvals for 69 MRs in an internal service using AI review
Key Questions Answered
What are the key details on Accurate components do not equal a reliable system?
A good example is SkinAgent AI, a skincare assistant that analyzes a photograph, clarifies the request, and recommends products. Its authors deliberately limited it to nonmedical recommendations: the system must not make diagnoses and must follow safety rules. The individual components looked convincing. The topical-request classifier achieved 99.84% accuracy.
What are the key details on Internal metrics can also be misleading?
Another study tested a common assumption: if the internal representations of text and an image are similar, the model must actually be using the image. The researchers replaced the image with random noise. Answer quality dropped sharply, but some alignment metrics continued to show high similarity. In other words, the instrument reported normal conditions even after the useful signal had disappeared.
What are the key details on A simulator is a testbed, not an acceptance test?
In robotics studies, coding agents wrote their own motion-planning programs and achieved success rates of 56–95% in several environments, compared with 47% for a conventional planner. This is a significant signal: an agent can experiment, clarify the rules of an environment, and transfer the program it discovers to new object layouts. But the tests were conducted primarily in simulation, where the agent could see the exact state of objects.
Delegate Deep Research, Coding & Multi-Step Workflows to Nomad Agent
Deploy autonomous task execution with Nomad Agent (agent.nomadnet.ai) — automated web research, data synthesis, code generation, and MCP tool orchestration.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.