Team trains an agent on other players' games and finishes 436th in Kaggriculture

A new version of the agent earned more money. Is it ready to ship to users? Not yet: it may simply have encountered a more favorable market. Kaggriculture participants faced exactly this problem. They compared versions on identical games, then trained a model to imitate strong players' decisions. This workflow is useful for agent developers: first establish a way to measure changes, then choose an architecture.

In a write-up published on Habr, the team reports finishing 436th among 10 148 teams and earning a silver medal. Over two months, the participants tested six approaches, from rules to a transformer with roughly 20 million parameters. By their assessment, the strongest model was a transformer trained through behavior cloning: given the game state, it learned to select actions from recorded games.

First, they separated agent quality from market luck

In Kaggriculture, two agents manage neighboring farms and sell their produce on a shared market. After 30 in-game days, whoever has the most money wins. If both produce a lot of milk, its price falls. If a yarn shop opens, it creates additional demand for wool. Producing more is not enough: agents need to understand what they can sell.

The assortment of shops is random and has a major impact on earnings. In the sample the team studied, the game itself accounted for 88.8% of the variance in final cash balances. This does not mean that chance determines 88.8% of wins: the metric describes the spread of the amounts earned.

To compare versions, the developers recreated games using the same random seed. The new agent replaced the old one, while the opponent replayed its recorded actions. They then counted how many previous losses became wins and how many wins were lost.

The method has an important limitation: the recorded opponent does not react to new behavior. This test setup helps screen changes, but its results cannot be presented as a win rate in a live tournament. There is a clear parallel for product teams: a fixed set of tasks is useful for comparing versions, but separate testing is needed to assess interaction with a changing environment.

Copying examples helped, but only for a while

The first imitation approach required no neural network training. The team collected 782 game recordings from one strong player. The agent searched for a similar situation based on object positions, shops and recent prices, then repeated the action from a matching recording.

This worked well early in the game, when openings often repeated. Later, game states diverged and suitable continuations became scarcer. When tested on the original player's exact game states, the copy reproduced that player's actions in 85% of cases on days 6–14, but in just 2.5% from day 15 onward.

This is a useful diagnostic for developers. An agent may confidently navigate the familiar beginning of a process, then lose its way after making several decisions of its own. A single average accuracy figure is therefore insufficient: developers should examine individual task stages and the points where suitable examples run out.

The neural network needed consistent decisions

The next step was a transformer. The farm state was represented by 197 tokens covering grid cells, workers, goods, the market and other entities. The model received information about both farms, while memory between turns helped it account for previous observations and actions.

Training revealed a labeling problem. Several equally short paths can lead to a cell, but a recording contains only one. The team modified the loss function so the model would not be penalized for taking another equally valid step toward the same goal. The practical lesson is simple: when several good decisions are acceptable, training should account for that.

Combining recordings from strong players did not guarantee improvement either. Their strategies involved different investments: an action that made sense after buying land might be unsuitable for a farm that had spent its money on animals. In early experiments, the model averaged these decisions and played worse.

This case offers a practical sequence for agent experiments: build a simple baseline, prepare repeatable scenarios, check that demonstrations are consistent and analyze errors by stage. A silver medal in a game simulation does not establish that the method is ready for business tasks. But the checks described help explain why another increase in model complexity actually worked—or merely looked convincing.