OpenGrall introduces “Engineer” mode: robot writes drivers and configures modules on its own

OpenGrall has described the "Engineer" mode — an architecture in which a robot can create plugins, write drivers, and adapt code to new hardware based on text commands. The idea is not about complete autonomy at all costs, but rather to delegate routine integration to an LLM while keeping human control.

How roles are structured

In the OpenGrall project, robot intelligence is divided into two circuits. The "Pilot" is a local model that runs onboard, makes decisions quickly, and manages movement in real time. It sees sensors, reads project files, and can update memory, but does not have the right to change system code.

This mode is needed so the robot doesn't depend on the cloud for basic tasks and doesn't break its own logic during normal operation. The "Engineer" activates only on request or when a task requires access to development tools. This is already a cloud LLM that is given access to project code, configurations, and documentation.

It doesn't just generate code blindly, but acts as a separate agent: studies the structure of plugins, searches for examples, edits files, runs code in a sandbox, and asks clarifying questions to the owner. If the command sounds like "turn left," the Pilot works; if it sounds like "configure the manipulator," the baton passes to the Engineer.

How the Engineer works

The main idea of the mode is not to write everything manually for each new module. If, for example, a servo camera on ESP32 is connected to the robot via WebSocket, the owner only needs to describe the device and available commands in a self-description file. After that, the Engineer reads the project, compares existing plugins, and creates a new module for specific hardware. OpenGrall has assembled a separate set of tools for this:

• searching for documentation and working examples on the web
• reading articles, specifications, and project files
• analyzing directory structure and existing plugins
• targeted code editing and creating new files
• running and testing the written code in an isolated sandbox

Next comes not "one-button magic," but a controlled development cycle. First, the model builds a class skeleton and can show it to a human before generating the full file. Then methods are filled in step by step in streaming mode to avoid regenerating the entire module and losing context.

For calibration, the Engineer can bring a human into the loop: ask them to place an object in front of the camera, measure the distance to the body, or confirm that the servo has reached the desired angle. This is especially important where a text description alone is not enough. The author bets not only on connecting new modules, but also on deeper system restructuring.

The article gives an example with a navigator: the basic manual version at approximately 1.5 thousand lines of code can be replaced with a much more extensive implementation with smooth trajectories, rerouting on the fly, and fewer jerks. In other words, we're no longer talking about generating a single driver, but about giving the robot a tool to rewrite its own subsystems for a new task.

"The final decision remains with the human."

Safety is maintained through three simple mechanisms: a sandbox for code execution, automatic backups before each change, and step-by-step generation with result verification. Code is executed with a timeout and without system calls, a file copy is created before editing, and integration into the main cycle does not happen automatically. Even if the model makes a mistake or goes down the wrong architectural path, a human can stop the process, roll back changes, and make the final decision manually.

Where the approach has limits

The OpenGrall authors directly say that the first generated code shouldn't be idealized. LLMs have the same chronic problems: outdated APIs, unnecessary dependencies, incorrect wrappers, and incompatibility with the environment. The difference is only that the agent can read logs, restart tests, and make corrections on its own until the module works. But in practice, such autonomous debugging can take not minutes, but hours of real time, especially if the hardware is non-standard or the documentation is rough.

There are also more rigid architectural limitations. For configuring a manipulator, camera, or lidar, this mode makes sense: the model sees the description, understands the geometry of the task, and can assemble a working interface on top of existing commands. But for controlling the gait of a hexapod or other tasks with strict requirements for speed and adaptability, self-generated Python code alone is not enough. According to the author, other approaches are needed there — for example, TinyML and learning in simulation, where behavior is developed through millions of iterations, not written superficially based on a text request.

What this means

OpenGrall demonstrates an interesting shift: a robot begins to be perceived not as a rigidly programmed device, but as a platform that can be further trained and extended with text commands. Complete autonomous robotics is still far away, but for integrating sensors, plugins, and individual subsystems, such an "Engineer" can already significantly reduce the amount of manual work — provided that the final switch still remains in human hands.