OpenGrall unveils an AI robot architecture where the language model handles strategy

OpenGrall proposes a simple but important shift in robotics: an LLM should not control a robot at the level of motors and instant reactions. Here the language model is responsible only for meaning, planning and choosing the next step, while safety, movement and low-level reflexes are moved into a separate loop. With this, the project tries to remove the main problem of most "robots with GPT" demos, where the machine talks nicely but then freezes for several seconds before every action.

The authors start with the most painful spot: safety. An LLM is non-deterministic: the same request can produce different answers, so trusting it with direct control of the motors is dangerous. OpenGrall introduces a hybrid scheme in which a TinyML model, or another rigid execution loop on a microcontroller, plays the role of a "spinal cord".

It knows the physics of the specific platform, manages the suspension and obstacle traversal, and has the right to veto any command. If the operator or the LLM says to drive forward, but the rangefinder sees an object closer than 10 centimeters, the command simply will not be executed. The claimed reaction time for an emergency stop is under 10 milliseconds.

The logic is thoroughly pragmatic: the LLM thinks, the executing module acts and can stop a dangerous action at any moment.

The second strength of OpenGrall is modularity. The project separates "thinking" from "doing" so that both parts can be swapped independently. The brain can be a local LLM, a VLM or a cloud model, if a harder task such as a multi-step plan or web search is needed.

The executor can be TinyML, a VLA model or even an ordinary hard-coded algorithm, if the platform is simple. The whole system is tied together through a WebSocket server, and devices connect as ordinary clients with roles such as agent, operator, lidar or esp. This makes it possible to add new sensors without rewriting the core, and even to build a setup where one agent works with several bodies at once: for example, a wheeled platform and a drone.

For an open source project on weak hardware this is an especially important emphasis: the architecture is not tied to one type of robot or one specific model.

The key engineering block is the SensorMemory and WeightCalculator pair. Instead of indiscriminately sending all raw streams to the LLM, the system collects data asynchronously, assesses its freshness and trustworthiness, and then turns it into a short prompt. If the lidar gets dirty or the VLM is blinded by sun, their weight drops before the decision is made. If one sensor is slow, it does not block the others. The article gives an illustrative example: the lidar point cloud is collapsed into eight sectors, and nearby objects are described by angle, distance, size and speed. For the LLM this is no longer noise but a structured situation.

An important nuance is that OpenGrall does not try to hand-write complex data fusion rules. The arbiter is the LLM itself, which sees the source, age and weight of each signal and on that basis chooses an action in JSON format. The system prompt is baked into the model in advance, for example via Ollama, so in the working loop only the "bare" operational part goes into the request. By the author's estimate, this shrinks the prompt from roughly 450 to 150 tokens.

A separate focus of the article is the fight against sluggishness. In many classic frameworks the robot waits for the slowest sensor, so fast telemetry effectively sits idle. OpenGrall abandons such synchronization: lidar, VLM and odometry write data to memory independently, and the agent takes the freshest and most reliable values at the current moment. But even then the LLM still thinks for hundreds of milliseconds, so inertial motion is added on the ESP32 side: if there is no new command yet, the robot does not freeze like a post but smoothly continues the last safe action with decaying speed.

Another optimization layer is caching decisions by context hash. If the robot again faces the same empty corridor, the system does not call the model again but takes the already verified decision from the cache. This idea is then extended toward habits and high-level reflexes: successful strategies can run without the LLM, and human feedback strengthens their weight. In addition, the project stores an episodic memory of human instructions and even allows autonomous goal-setting, where an idle robot chooses by itself what to explore, what to remember or whom to start a dialogue with.

Seen more broadly, OpenGrall is interesting not as yet another attempt to "bolt GPT onto a robot" but as an attempt to bring LLM robotics to a more mature architecture. It makes no promise of a magical universal brain, but it offers a clear separation of responsibilities, work with limited hardware, protection from dangerous actions and a path to gradual learning without retraining the whole system. For developers this means a more realistic way to build robots on modern models: use the LLM where it is strong, and do not force it to do what a small, fast and predictable execution loop does better.