RWE-bench Tests AI on Engineering Tasks: Best Result Is 37%

A code generator easily impresses in a demo: you give it a task, get a function, run the tests. With an engineering design, that trick breaks down quickly. A crankshaft can look plausible but fail from fatigue. A printed circuit board can look neat on screen but fail to meet constraints on delays and trace resistance.

The startup Xplore built RWE-bench, a set of 20 tasks in mechanics, electronics and aerospace engineering. Here the model is expected to deliver not an explanation or a sketch "for inspiration", but a finished result that can be verified by calculations. The best reported average result is 37%. It is an important cold shower for anyone who has already mentally replaced the engineering department with an agent that has CAD access.

How the verification works

Imagine a task: design a one-piece steel crankshaft for a three-cylinder diesel engine. It must fit the specified bearings and flywheel, clear the connecting rods, withstand full load at 2,400 rpm and retain a service life of millions of cycles. It is not enough to draw a suitable shape: the design must simultaneously meet constraints on mass, stress, stiffness, balancing and bearing load.

Other tasks follow a similar pattern. In one, the model routes a 71-net DDR3L interface between a Zynq-7000 processor and two memory chips on an already populated eight-layer board. In another, it builds a mesh to compute flow around a turbine blade, achieves convergence and calculates the heat flux.

The model gets a sandbox with CadQuery, Gmsh, CalculiX and Python. It can create geometry, run physics calculations and fix its solution. The final work is accepted by a deterministic program: it re-checks the requirements and runs the solvers again. There are no judge models or manual grading in this loop. If even one mandatory constraint is violated, the task does not count.

What the results showed

According to Xplore, GPT-6 Astra solved 37% of tasks on average at a cost of $58.83 per run. Claude Opus 5.5 scored 29% at $57.52, and GPT-6.1 Sol 25% at $10.28. The latter looks more practical in terms of price versus result: it trailed the leader by 12 percentage points but cost almost six times less.

The other participants lagged noticeably. Gemini 4 Argon got 12%, Claude Fable 5.1 7%, and Kimi K3.1 Pro 1%. Five models solved no tasks at all. A high price did not guarantee quality either: a Fable run cost $97.57 on average.

However, it is too early to turn the table into a definitive ranking of models. The set contains only 20 tasks, the tasks and the verification program are closed, and testing is done by arrangement with Xplore. The company also sells training environments built on a similar methodology. This does not prove the results are incorrect, but it creates a commercial conflict of interest and hinders independent reproduction. The public EngiWorld from Chinese universities is more transparent: its code, data and methodology description are available, although its tasks and conditions differ.

What teams should do with this

The main value of RWE-bench is not yet another leaderboard. It shows the right architecture for an engineering agent: the model works inside a tool loop, runs calculations itself, and the result is accepted by independent verification code.

If you are building a similar system, start not by choosing the smartest model but by formalizing requirements. Translate tolerances, loads, geometric constraints and quality criteria into automated checks. Give the agent access to calculation tools, keep intermediate artifacts and do not release a result without reproducible validation.

At 37% successful solutions, such an agent has not yet become an autonomous designer. But as an executor that quickly iterates through options under strict control, it can already be useful. The practical conclusion is boring but workable: what deserves trust is not the model's reasoning but the system of checks around it.