Live Nov 18 · Cohort 3Nov 18, 19:00 MSK — free hands-on AI agents live workshop · Live Cohort 3 starts Nov 23Join Nov 18 Live →Cohort 3 (Nov 23)
ReleaseInsight · On RadarHabr AI→ original

Local Qwen scores 92.3% on lab log searches and 40.9% on equipment control

Good document retrieval does not yet mean an agent can be trusted with equipment. In an experiment described on Habr, a locally run Qwen scored 92.3% on laboratory log tasks, but only 40.9% on an equipment control test bench. For example, it described checking the text of a process recipe as a completed run. For developers, this is a reason to test retrieval, long sequences of actions, and compliance with access permissions separately.

AI-processed from Habr AI; edited by Hamidun News
Local Qwen scores 92.3% on lab log searches and 40.9% on equipment control
Source: Habr AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
Key metric #1Qwen scored 62.5% on X-ray optics tasks.
Key metric #2Correcting the instruction manual raised Qwen's score from 47.5% to 62.5%.
Key metric #3Fixing tool calls did not eliminate errors in its conclusions.

The agent found the right database entry, extracted the numbers correctly, and delivered a convincing answer. Is it ready to be connected to operational tools? In a laboratory experiment described on Habr, a locally run Qwen performed well on logbook tasks, but considerably worse on calculations and equipment control. In one case, it checked the text of a process recipe, read the readings from a stationary chamber, and described everything as though deposition had already taken place.

This is a useful case study for teams building agents on top of internal documents and APIs. It shows which errors persist after a model has learned to find data and format function calls correctly. All the results cited are the article authors' findings from their own test benches, rather than a universal ranking of models.

Three tasks, three different results

The authors tested six models on 68 tasks: searching and cross-checking laboratory log entries, reconstructing mirror structures from X-ray measurements, and working with a vacuum system emulator.

The local Qwen scored 24 out of 26 points, or 92.3%, on the logbook tasks. It found typos, spotted unit mix-ups, and matched entries to measurement files. But an unwelcome habit was already apparent: turning ambiguous information into confident assertions. If a sample name inside a file conflicted with the filename and revision history, the model could decide for itself which sample the file belonged to.

On X-ray reflectometry tasks, it scored 62.5%. These tasks require reconstructing the thicknesses and other parameters of a mirror's layers from a measured curve. Several different structures can produce similar curves. A good numerical fit therefore still needs verification: the model must look for alternatives and identify parameters that cannot be determined unambiguously.

On the equipment control test bench, Qwen scored 9 out of 22 points, or 40.9%, finishing last among the participants. It did, however, successfully complete one large deposition task in the emulator. Its poor overall performance stemmed from individual fabrications, incorrect conclusions, and violations of specified constraints.

Reliable function calls do not guarantee correct decisions

The model ran on a computer with an RTX 5080 with 16 GB of VRAM and 64 GB of RAM. According to the authors, the quantized weights occupied 46.84 GiB. A headline referring to 16 GB of VRAM therefore does not mean the entire model fit on the graphics card.

Long tasks required additional configuration. Sessions grew to 180–260 thousand tokens, the model got stuck in loops over numerical tables, and the parser sometimes concatenated regenerated function arguments. The authors increased the available context, capped the reasoning budget, and changed how tool calls were handled.

After those fixes, they reported no protocol errors across more than four thousand calls. Substantive mistakes remained, however. Qwen could accept the first plausible solution, skip a mandatory check, or forget some constraints when completing its final report.

For a product, these are separate quality criteria. A function can receive valid arguments and execute successfully, only for the agent to explain the result incorrectly afterward. In the experiment described, a validator accepted an arbitrary name for a shutter position. The model spotted this gap but still declared the original recipe workable.

Test both the agent and your own instruction manual

One of the most useful episodes concerns the authors' own mistakes. Several models failed the same difficult tasks. An investigation uncovered four defects in the instructions, settings, and acceptance criteria, ranging from an omitted material to a requirement to check an unnecessary sublayer.

After the corrections, Qwen's score on the optics test bench rose from 47.5% to 62.5%. This improvement came without changing the model. There is a concrete task here for development teams: investigate failures shared by several models and check whether the instructions, available tools, and evaluation criteria are aligned.

The sample is small, the test benches are specialized, and the supplied text cuts off before the analysis is complete. It cannot establish a general ceiling on the capabilities of local models or isolate the effects of quantization from other settings. But its testing approach can be applied to your own product: evaluate retrieval accuracy, performance on long sequences of actions, and compliance with permissions separately. And always check whether the agent distinguishes between finding a record, successfully validating something, and actually carrying out an operation.

Story

Key Questions Answered

What are the key details on Three tasks, three different results?

The authors tested six models on 68 tasks: searching and cross-checking laboratory log entries, reconstructing mirror structures from X-ray measurements, and working with a vacuum system emulator. The local Qwen scored 24 out of 26 points, or 92.3%, on the logbook tasks. It found typos, spotted unit mix-ups, and matched entries to measurement files. But an unwelcome habit was already apparent: turning ambiguous information into confident assertions.

What are the key details on Reliable function calls do not guarantee correct decisions?

The model ran on a computer with an RTX 5080 with 16 GB of VRAM and 64 GB of RAM. According to the authors, the quantized weights occupied 46.84 GiB. A headline referring to 16 GB of VRAM therefore does not mean the entire model fit on the graphics card. Long tasks required additional configuration. Sessions grew to 180–260 thousand tokens, the model got stuck in loops over numerical tables, and the parser sometimes concatenated regenerated function arguments.

What are the key details on Test both the agent and your own instruction manual?

One of the most useful episodes concerns the authors' own mistakes. Several models failed the same difficult tasks. An investigation uncovered four defects in the instructions, settings, and acceptance criteria, ranging from an omitted material to a requirement to check an unnecessary sublayer. After the corrections, Qwen's score on the optics test bench rose from 47.5% to 62.5%. This improvement came without changing the model.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…