Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)

DeepSeek Flash Handles Delphi Tasks 12x Cheaper Than Sonnet 5.5 and GPT-6.1 Sol

Public leaderboards don't help when choosing an agent for Delphi, so the developer of lab software ran seven models on 20 tasks from his closed repositories. DeepSeek V4.1 Flash cost 12 times less than Sonnet 5.5 and GPT-6.1 Sol: those two cost about $10 for the full set, or $0.46–$0.55 per task. But Sonnet 5.5 lost one task to an extra compiler warning. Evaluate the agent on your own code, and count the warnings.

AI-processed from Хабр: Машинное обучение; edited by Hamidun News
DeepSeek Flash Handles Delphi Tasks 12x Cheaper Than Sonnet 5.5 and GPT-6.1 Sol
Source: Хабр: Машинное обучение. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceХабр: Машинное обучение ↗
Editor-in-Chief
Terra$3.52–$3.56 for 20 tasks, a third of the cost of Sol and Sonnet
Key metric #2Sol spent 7–11 minutes per task and twice hit the 45-minute limit
Key metric #3In XRC-06 all 37 tests passed, but there were 2 warnings instead of 1

If you write in Delphi, public leaderboards for coding models barely help: they test different tasks, and the leaders regularly break VCL forms. The author, who has spent years building scientific and laboratory software in Delphi, decided to test agents on his own code. He took 20 real tasks from three closed repositories and ran them through seven cloud and local models.

The main cost finding: DeepSeek V4.1 Flash completes the full set at roughly 12 times less than Sonnet 5.5 and GPT-6.

1 Sol. For the latter two, the full set of 20 tasks costs about $10, or $0.46–$0.

55 per task. The main quality finding is more surprising: even a strong model can score zero on a task where all tests are green.

How the test works

The repositories are X-Ray Calc 3, an X-ray reflectometry program; software for a tribometer; and a client for an electronic lab journal with an MCP server. All three are closed, so the models' training data contains no ready-made solutions. A script went through the history and found 1,547 commits with bug fixes and minor improvements: 831 in the journal, 443 in X-Ray Calc 3, and 273 in the tribometer software. From these, the author selected 20 tasks: 11 on logic, calculations and data flows, and 9 on VCL forms, tables, charts and dialogs.

The task examples will be clear to any developer. Numbers in JSON were rounded by a function that could not handle very small values, and film profile fitting crashed with an error. The axis boundary field on a chart was recalculated on every keystroke, so while typing 1e-8, intermediate input such as "1" or "1e" was treated as a complete number. The database tool crashed on nested FireDAC transactions. And the Help menu needed a check for updates on GitHub with SHA-256 checksum verification.

Each task was checked with the RAD Studio 37.0 compiler: before the fix, the project builds but the new DUnitX tests fail; after the author's fix, everything builds and the tests pass. The console agent received a clean copy of the repository, a task description, and the build command.

Hint files for agents were removed, the language server was disabled, and the limit was set to 45 minutes. There were two modes: without instructions, and with the delphi-development skill, which describes project structure and four rules. Among them: visual components only in .

dfm files, the TImageCollection and TVirtualImageList pair instead of TImageList, and freeing local objects with try..finally. A task does not count as passed if the number of compiler warnings rises above the original level.

Who showed what

DeepSeek Flash and Claude Haiku 5.5 were run three times per task; the other models were run once. All cloud models ran at high reasoning levels. DeepSeek Flash with the skill passed all tests in 60 of 60 runs, and on the strict scale scored 58 of 60. It takes an average of 337–383 seconds per task. It also outscored Haiku 5.5, which is close in price: Haiku costs $0.061 per task and succeeds in 83–88% of runs.

GPT-6.1 Terra was released the day before publication. At OpenAI's rates it costs $1, $5 and $0.05 per million input, output and cached tokens, half the price of Sol. The full set cost it $3.52–$3.56, a third of what Sol and Sonnet cost. On tests, Terra scored 19 of 20 (95%). Terra solved the auto-update task XRC-09 on the first attempt, a task where Sol failed in both modes. Sol usually spends 7–11 minutes on a task, and on two journal tasks without the skill it used the full 45-minute limit.

Not included in the table: Gemini 4 Argon (stopped after 24 runs because of its speed), GLM 5.3 Flash (its OpenRouter balance ran out after the fourth task), and Claude Opus 5.5 (two trial tasks, since Sonnet already handled them).

How one warning sank almost everyone

The most telling story is task XRC-06: add a button to clear results from a project file that is, internally, a plain zip archive. Sonnet 5.5 passed all 20 tasks on tests, but scored 19 of 20 on the strict scale.

The cause was not the logic. All 37 hidden tests passed, but the project already had one compiler warning, W1002 ("symbol is platform-dependent"), in an existing archive-writing function. Sol and DeepSeek's first attempt moved the shared logic into a helper procedure, and the warning count stayed at one.

Sonnet, Terra, Haiku, Qwen Coder and two other DeepSeek attempts wrote a separate function and copied the read-only check from a neighboring function. That made two lines with the problematic symbol, so the warning count became two instead of one, and the script gave a zero.

The second lesson comes from task TRB-04. The window crashed on opening because of a pair of TStringGrid properties: the row count has to be increased first, and only then the fixed row count set. The error in FormCreate displayed a modal Windows dialog that nobody could dismiss by clicking "OK", so the test hung until timeout instead of simply failing.

What to do in practice

Don't choose an agent by its public rating. Assemble 10–20 tasks with tests from your own commit history. Include the compiler warning count in the evaluation, or you will miss exactly the kind of failure that broke Sonnet here. For routine fixes, the cheap DeepSeek Flash looks more sensible than models that cost $10 per set. The obvious limitation: 20 tasks and one author make this a reference point, not statistics that apply to all projects.

Story

Key Questions Answered

How the test works?

The repositories are X-Ray Calc 3, an X-ray reflectometry program; software for a tribometer; and a client for an electronic lab journal with an MCP server. All three are closed, so the models' training data contains no ready-made solutions. A script went through the history and found 1,547 commits with bug fixes and minor improvements: 831 in the journal, 443 in X-Ray Calc 3, and 273 in the tribometer software.

Who showed what?

DeepSeek Flash and Claude Haiku 5.5 were run three times per task; the other models were run once. All cloud models ran at high reasoning levels. DeepSeek Flash with the skill passed all tests in 60 of 60 runs, and on the strict scale scored 58 of 60. It takes an average of 337–383 seconds per task. It also outscored Haiku 5.5, which is close in price: Haiku costs $0.061 per task and succeeds in 83–88% of runs. GPT-6.1 Terra was released the day before publication.

How one warning sank almost everyone?

The most telling story is task XRC-06: add a button to clear results from a project file that is, internally, a plain zip archive. Sonnet 5.5 passed all 20 tasks on tests, but scored 19 of 20 on the strict scale. The cause was not the logic. All 37 hidden tests passed, but the project already had one compiler warning, W1002 ("symbol is platform-dependent"), in an existing archive-writing function. Sol and DeepSeek's first attempt moved the shared logic into a helper procedure, and the warning count stayed at one.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…