LLM Controls an Online RPG Hero: Gemini Can Only Handle Every 10th Game Step

Most game projects involving LLMs end up as conversations: the neural network plays an NPC and answers lines of dialogue. The author of this experiment went further and handed the model control of a character in a networked RPG. The difference is fundamental: you can say anything, but you can only do what the rules of the world allow.

How it works

The game is asynchronous: the server doesn't wait for the player, and other characters and goblins run around. That's why the client is built on three threads. The first receives lines from the server and parses them into a structure. The second sends a request to the LLM and waits for the response. The third relays the picture to a browser so a human can watch what is happening.

The prompt is assembled on the fly from observations: what the character sees and hears, what is in its inventory, which resources are needed for the goal, and which actions are currently available. At the start, a static MASTER-PROMPT with the game rules and crafting recipes is sent. After that, each response is linked to the next request via an identifier, like a continuation of a dialogue.

The response format is defined by a response_format schema. The author compared two approaches: numeric fields with a probability for each action, and a single action field with an enum. He found no noticeable difference in quality and chose the enum because it is easier to parse. In addition, the model returns a goal, plans, thoughts and an optional line of speech.

Where everything runs into limits

On the free tier, lite models are allowed one request every 4 seconds. The server step is 0.45 s, so at best the LLM sees every tenth step. The daily limit is 500 requests, after which you have to switch models. There is also a non-obvious limit: the response chain breaks with a maximum-depth error after roughly 100 iterations. It may depend on how full the context is.

The agent's memory had to be built by hand: a cell-based memory of important objects, an event memory and a memory of past actions. A separate scarcity memory appeared as well. The model confidently "puts into the smelter" items it doesn't have. When the parser notices the mismatch, the missing item is recorded, and actions that contradict it are filtered out.

What came out of it

The agent walks, collects items, builds structures and occasionally smelts metal. Once it cast a pickaxe head, one of the most complex recipes, although it found some of the ingredients on a dwarf's corpse. Its lines are quite meaningful; it comments on its actions and tries to talk to NPCs.

But the behavior can't be called intelligent. The agent falls into dead loops: it crafts unneeded things, spins in place and gets stuck in trades. It can ignore the hint "there is a wall with the needed minerals on the right" and walk past. Its only successful deal with a dwarf turned out to be a losing one: it traded all its tin for copper ore, coal and clay, that is, the rare for the common.

Practical takeaway

First, speed is decisive. If a response arrived in half a second, the game could run synchronously without losing steps. The author is considering a small local model. Second, a list of coordinates in text works poorly as a spatial description. Third, the most sensible conclusion is this: the LLM should be given high-level decisions and speech, such as "go to the forge", while pathfinding and execution should be handed to classical algorithms.

This is the same pattern as in production agents: the model plans, deterministic code executes and verifies. The experiment's limitation is that it is a single naive prototype without metrics, but the set of pitfalls with limits, looping and memory will be familiar to anyone building agents.