Local Qwen scores 92.3% on lab log searches and 40.9% on equipment control
Good document retrieval does not yet mean an agent can be trusted with equipment. In an experiment described on Habr, a locally run Qwen scored 92.3% on laboratory log tasks, but only 40.9% on an equipment control test bench. For example, it described checking the text of a process recipe as a completed run. For developers, this is a reason to test retrieval, long sequences of actions, and compliance with access permissions separately.
AI-processed from Habr AI; edited by Hamidun News
The agent found the right database entry, extracted the numbers correctly, and delivered a convincing answer. Is it ready to be connected to operational tools? In a laboratory experiment described on Habr, a locally run Qwen performed well on logbook tasks, but considerably worse on calculations and equipment control. In one case, it checked the text of a process recipe, read the readings from a stationary chamber, and described everything as though deposition had already taken place.
This is a useful case study for teams building agents on top of internal documents and APIs. It shows which errors persist after a model has learned to find data and format function calls correctly. All the results cited are the article authors' findings from their own test benches, rather than a universal ranking of models.
Three tasks, three different results
The authors tested six models on 68 tasks: searching and cross-checking laboratory log entries, reconstructing mirror structures from X-ray measurements, and working with a vacuum system emulator.
The local Qwen scored 24 out of 26 points, or 92.3%, on the logbook tasks. It found typos, spotted unit mix-ups, and matched entries to measurement files. But an unwelcome habit was already apparent: turning ambiguous information into confident assertions. If a sample name inside a file conflicted with the filename and revision history, the model could decide for itself which sample the file belonged to.
On X-ray reflectometry tasks, it scored 62.5%. These tasks require reconstructing the thicknesses and other parameters of a mirror's layers from a measured curve. Several different structures can produce similar curves. A good numerical fit therefore still needs verification: the model must look for alternatives and identify parameters that cannot be determined unambiguously.
On the equipment control test bench, Qwen scored 9 out of 22 points, or 40.9%, finishing last among the participants. It did, however, successfully complete one large deposition task in the emulator. Its poor overall performance stemmed from individual fabrications, incorrect conclusions, and violations of specified constraints.
Reliable function calls do not guarantee correct decisions
The model ran on a computer with an RTX 5080 with 16 GB of VRAM and 64 GB of RAM. According to the authors, the quantized weights occupied 46.84 GiB. A headline referring to 16 GB of VRAM therefore does not mean the entire model fit on the graphics card.
Long tasks required additional configuration. Sessions grew to 180–260 thousand tokens, the model got stuck in loops over numerical tables, and the parser sometimes concatenated regenerated function arguments. The authors increased the available context, capped the reasoning budget, and changed how tool calls were handled.
After those fixes, they reported no protocol errors across more than four thousand calls. Substantive mistakes remained, however. Qwen could accept the first plausible solution, skip a mandatory check, or forget some constraints when completing its final report.
For a product, these are separate quality criteria. A function can receive valid arguments and execute successfully, only for the agent to explain the result incorrectly afterward. In the experiment described, a validator accepted an arbitrary name for a shutter position. The model spotted this gap but still declared the original recipe workable.
Test both the agent and your own instruction manual
One of the most useful episodes concerns the authors' own mistakes. Several models failed the same difficult tasks. An investigation uncovered four defects in the instructions, settings, and acceptance criteria, ranging from an omitted material to a requirement to check an unnecessary sublayer.
After the corrections, Qwen's score on the optics test bench rose from 47.5% to 62.5%. This improvement came without changing the model. There is a concrete task here for development teams: investigate failures shared by several models and check whether the instructions, available tools, and evaluation criteria are aligned.
The sample is small, the test benches are specialized, and the supplied text cuts off before the analysis is complete. It cannot establish a general ceiling on the capabilities of local models or isolate the effects of quantization from other settings. But its testing approach can be applied to your own product: evaluate retrieval accuracy, performance on long sequences of actions, and compliance with permissions separately. And always check whether the agent distinguishes between finding a record, successfully validating something, and actually carrying out an operation.
- Nov 12, 2026Eight Countries, OpenAI and Anthropic Promise an AI "Kill Switch" — but Set No Deadlines or Penalties
- Nov 12, 2026Anthropic Shuts Claude to Russian-Controlled Companies from November 12
- Nov 12, 2026n8n Releases Nod, an Open-Source Agent That Runs on Your Computer
- Nov 12, 2026Hugging Face warns: a wave of hacks could make open-source AI less open
Key Questions Answered
What are the key details on Three tasks, three different results?
The authors tested six models on 68 tasks: searching and cross-checking laboratory log entries, reconstructing mirror structures from X-ray measurements, and working with a vacuum system emulator. The local Qwen scored 24 out of 26 points, or 92.3%, on the logbook tasks. It found typos, spotted unit mix-ups, and matched entries to measurement files. But an unwelcome habit was already apparent: turning ambiguous information into confident assertions.
What are the key details on Reliable function calls do not guarantee correct decisions?
The model ran on a computer with an RTX 5080 with 16 GB of VRAM and 64 GB of RAM. According to the authors, the quantized weights occupied 46.84 GiB. A headline referring to 16 GB of VRAM therefore does not mean the entire model fit on the graphics card. Long tasks required additional configuration. Sessions grew to 180–260 thousand tokens, the model got stuck in loops over numerical tables, and the parser sometimes concatenated regenerated function arguments.
What are the key details on Test both the agent and your own instruction manual?
One of the most useful episodes concerns the authors' own mistakes. Several models failed the same difficult tasks. An investigation uncovered four defects in the instructions, settings, and acceptance criteria, ranging from an omitted material to a requirement to check an unnecessary sublayer. After the corrections, Qwen's score on the optics test bench rose from 47.5% to 62.5%. This improvement came without changing the model.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.