DeepSeek Flash Handles Delphi Tasks 12x Cheaper Than Sonnet 5.5 and GPT-6.1 Sol
Public leaderboards don't help when choosing an agent for Delphi, so the developer of lab software ran seven models on 20 tasks from his closed repositories. DeepSeek V4.1 Flash cost 12 times less than Sonnet 5.5 and GPT-6.1 Sol: those two cost about $10 for the full set, or $0.46–$0.55 per task. But Sonnet 5.5 lost one task to an extra compiler warning. Evaluate the agent on your own code, and count the warnings.
AI-processed from Хабр: Машинное обучение; edited by Hamidun News
If you write in Delphi, public leaderboards for coding models barely help: they test different tasks, and the leaders regularly break VCL forms. The author, who has spent years building scientific and laboratory software in Delphi, decided to test agents on his own code. He took 20 real tasks from three closed repositories and ran them through seven cloud and local models.
The main cost finding: DeepSeek V4.1 Flash completes the full set at roughly 12 times less than Sonnet 5.5 and GPT-6.
1 Sol. For the latter two, the full set of 20 tasks costs about $10, or $0.46–$0.
55 per task. The main quality finding is more surprising: even a strong model can score zero on a task where all tests are green.
How the test works
The repositories are X-Ray Calc 3, an X-ray reflectometry program; software for a tribometer; and a client for an electronic lab journal with an MCP server. All three are closed, so the models' training data contains no ready-made solutions. A script went through the history and found 1,547 commits with bug fixes and minor improvements: 831 in the journal, 443 in X-Ray Calc 3, and 273 in the tribometer software. From these, the author selected 20 tasks: 11 on logic, calculations and data flows, and 9 on VCL forms, tables, charts and dialogs.
The task examples will be clear to any developer. Numbers in JSON were rounded by a function that could not handle very small values, and film profile fitting crashed with an error. The axis boundary field on a chart was recalculated on every keystroke, so while typing 1e-8, intermediate input such as "1" or "1e" was treated as a complete number. The database tool crashed on nested FireDAC transactions. And the Help menu needed a check for updates on GitHub with SHA-256 checksum verification.
Each task was checked with the RAD Studio 37.0 compiler: before the fix, the project builds but the new DUnitX tests fail; after the author's fix, everything builds and the tests pass. The console agent received a clean copy of the repository, a task description, and the build command.
Hint files for agents were removed, the language server was disabled, and the limit was set to 45 minutes. There were two modes: without instructions, and with the delphi-development skill, which describes project structure and four rules. Among them: visual components only in .
dfm files, the TImageCollection and TVirtualImageList pair instead of TImageList, and freeing local objects with try..finally. A task does not count as passed if the number of compiler warnings rises above the original level.
Who showed what
DeepSeek Flash and Claude Haiku 5.5 were run three times per task; the other models were run once. All cloud models ran at high reasoning levels. DeepSeek Flash with the skill passed all tests in 60 of 60 runs, and on the strict scale scored 58 of 60. It takes an average of 337–383 seconds per task. It also outscored Haiku 5.5, which is close in price: Haiku costs $0.061 per task and succeeds in 83–88% of runs.
GPT-6.1 Terra was released the day before publication. At OpenAI's rates it costs $1, $5 and $0.05 per million input, output and cached tokens, half the price of Sol. The full set cost it $3.52–$3.56, a third of what Sol and Sonnet cost. On tests, Terra scored 19 of 20 (95%). Terra solved the auto-update task XRC-09 on the first attempt, a task where Sol failed in both modes. Sol usually spends 7–11 minutes on a task, and on two journal tasks without the skill it used the full 45-minute limit.
Not included in the table: Gemini 4 Argon (stopped after 24 runs because of its speed), GLM 5.3 Flash (its OpenRouter balance ran out after the fourth task), and Claude Opus 5.5 (two trial tasks, since Sonnet already handled them).
How one warning sank almost everyone
The most telling story is task XRC-06: add a button to clear results from a project file that is, internally, a plain zip archive. Sonnet 5.5 passed all 20 tasks on tests, but scored 19 of 20 on the strict scale.
The cause was not the logic. All 37 hidden tests passed, but the project already had one compiler warning, W1002 ("symbol is platform-dependent"), in an existing archive-writing function. Sol and DeepSeek's first attempt moved the shared logic into a helper procedure, and the warning count stayed at one.
Sonnet, Terra, Haiku, Qwen Coder and two other DeepSeek attempts wrote a separate function and copied the read-only check from a neighboring function. That made two lines with the problematic symbol, so the warning count became two instead of one, and the script gave a zero.
The second lesson comes from task TRB-04. The window crashed on opening because of a pair of TStringGrid properties: the row count has to be increased first, and only then the fixed row count set. The error in FormCreate displayed a modal Windows dialog that nobody could dismiss by clicking "OK", so the test hung until timeout instead of simply failing.
What to do in practice
Don't choose an agent by its public rating. Assemble 10–20 tasks with tests from your own commit history. Include the compiler warning count in the evaluation, or you will miss exactly the kind of failure that broke Sonnet here. For routine fixes, the cheap DeepSeek Flash looks more sensible than models that cost $10 per set. The obvious limitation: 20 tasks and one author make this a reference point, not statistics that apply to all projects.
- Nov 21, 2026Free Claude Code Runs Claude Code on DeepSeek and Qwen: 60+ Providers, 68,000 Stars
- Nov 21, 2026GPT-4o Pulled After 13 Lawsuits: AI Ethics Is Sold With the Model, and It Costs Dearly
- Nov 21, 2026Linger Runs 12 AI Models on Your PC: Up to 5x Faster and $0 in Token Costs
- Nov 21, 2026Claude Opus 5.5 Picked a School in Iran as a Military Target in 22 of 32 Tries. OpenAI Models Never Did.
Key Questions Answered
How the test works?
The repositories are X-Ray Calc 3, an X-ray reflectometry program; software for a tribometer; and a client for an electronic lab journal with an MCP server. All three are closed, so the models' training data contains no ready-made solutions. A script went through the history and found 1,547 commits with bug fixes and minor improvements: 831 in the journal, 443 in X-Ray Calc 3, and 273 in the tribometer software.
Who showed what?
DeepSeek Flash and Claude Haiku 5.5 were run three times per task; the other models were run once. All cloud models ran at high reasoning levels. DeepSeek Flash with the skill passed all tests in 60 of 60 runs, and on the strict scale scored 58 of 60. It takes an average of 337–383 seconds per task. It also outscored Haiku 5.5, which is close in price: Haiku costs $0.061 per task and succeeds in 83–88% of runs. GPT-6.1 Terra was released the day before publication.
How one warning sank almost everyone?
The most telling story is task XRC-06: add a button to clear results from a project file that is, internally, a plain zip archive. Sonnet 5.5 passed all 20 tasks on tests, but scored 19 of 20 on the strict scale. The cause was not the logic. All 37 hidden tests passed, but the project already had one compiler warning, W1002 ("symbol is platform-dependent"), in an existing archive-writing function. Sol and DeepSeek's first attempt moved the shared logic into a helper procedure, and the warning count stayed at one.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.