BrickWright Teaches Coding Agents to Build with LEGO: 100% Valid Builds vs. 9–69%

The BrickWright framework makes off-the-shelf coding agents build LEGO models that can actually be assembled by hand. On a set of 512 prompts, agents running under its control produced 100% physically valid builds. The same models without it did noticeably worse: GPT-6 Luna managed only 9%, and the best baseline agent, GPT-6 Astra, reached 69%.

Why agents build out of thin air

Modern coding agents handle 3D modeling with confidence, but the physical world is less forgiving. A virtual object can be drawn so that some parts hang with no support, others pass straight through their neighbors, and still others end up as a separate island. On screen this is sometimes hard to spot; in a box of bricks it becomes a defect.

The authors define physical feasibility strictly. A build is valid if it contains only real parts, has no overlapping or "floating" elements, and forms a single connected structure whenever that is possible at all. Their thesis: to get such builds, an agent does not need fine-tuning. It is enough to give it tools that assemble the model and check its physics.

BrickLayer: a convenient API and a validator in the loop

BrickWright controls the agent through code. The agent writes programs in BrickLayer, a Python toolkit with two main parts.

The first is an ergonomic API for building from bricks. It cuts inference cost in half compared with working in raw LDraw format, where the agent manually handles a low-level description of the parts. Fewer tokens spent on routine work means each attempt is cheaper.

The second part is a validator. It checks the build and returns feedback to the agent: what is hanging, what overlaps, what is not connected to the rest. The agent sees the errors as text and fixes its code until the structure is valid.

Key figures from the paper:

• 100% physical validity on a set of 512 prompts for agents using BrickWright
• 9% for GPT-6 Luna in the baseline configuration
• 69% for GPT-6 Astra in the baseline configuration
• 2× inference cost savings from the BrickLayer API versus raw LDraw
• input: a text or image prompt, running on pretrained agents with no separate training

Validity did not eat into quality

Reliability is easy to buy at the price of boring results: an agent could build only flat plates, and they would pass any check. That is why the second half of the result matters. For lightweight agents, prompt alignment improved; for strong agents, it stayed at the same level. In other words, the physical constraints did not force the models to simplify their designs.

Separately, the authors compared the method with specialized text-to-brick models, which are trained specifically to generate brick builds. The BrickWright agents outperformed them. A general-purpose model with good tools turned out to be stronger than a narrow solution.

Where this approach is useful

The direct application is clear: generating LEGO instructions and sets from a description or photo, where the output cannot be a structure that collapses on the table. The project code is open, so you can take it and plug it into your own agent.

The broader value lies in the pattern itself: a domain-specific API plus a validator that feeds errors back into the loop. It suits any task where the result must obey strict rules: CAD parts, equipment layout, assembly from standard components. The takeaway for agent developers is simple. Before improving the model, check whether it has a proper ruler at hand for self-checking. Here, that ruler lifted the worst result from 9% to 100% and also cut token costs in half.