JockeyBench: 28 LLM Agent Configurations All Lost Money on Japanese Horse Racing

If an LLM agent predicts the outcome of an event reasonably well, can you make money with it? The HKUST KnowComp team tested this on Japanese horse racing and got an uncomfortable answer. In the JockeyBench benchmark, 28 "model plus scaffold" configurations were run on 324 historical races. After correction for multiple comparisons, paired tests found seven improvements from revising the forecast and two degradations. Yet average profit under capped-Kelly betting is negative for every configuration, without exception.

Horse racing as a testbed for agents

The authors (Zhongwei Xie, Xingsu Li, Jingqiao Yu, Zihao Wang, Hong ChiTat, Zhaowei Wang, Weizhi Du, Yuxuan Liu, Huihao Jing, Jiaxin Bai and Yangqiu Song) are tackling a specific problem: repeated event forecasting. An agent has to extract a signal from noisy data, then show that the signal is worth something across a series of sequential decisions. Horse racing suits this kind of test almost perfectly: events are regular, outcomes are unambiguous, and the betting market puts a price on being wrong.

The key engineering idea is how the experiment is kept honest. Data is released in time order, and the agent sees nothing from the future relative to the race. Tools cost money, so research has a price and counts in the final score. For comparison, the benchmark uses paired controls with frozen programs, which separates the effect of revising a forecast from random variation.

Two workflows and three layers of evaluation

The benchmark compares two workflows:

• the LLM writes a predictor, a program that is then executed and produces forecasts;
• the LLM gathers evidence and makes a fresh forecast for each individual race.

Evaluation is split into three independent levels:

• forecast quality;
• net utility specific to the benchmark, which accounts for tool fees;
• retrospective return and betting risk under fixed decision rules.

The description of the work does not name the specific models, so it cannot be used to compare vendors. There is, however, a structural finding. In a comparison of four models, the reusable-program mode delivers higher average profit and lower research spending. The gain is relative: on average, both modes lose money.

Where the money hides and where it disappears

The diagnostics are the most interesting part. The average probability bias, which looks tolerable across the whole sample, flips sign on the bets the agent actually chose. In other words, the model is especially confident where its confidence is wrong: bet selection amplifies overestimates. This is a familiar affliction of any system that hunts for "value" relative to the market, and LLMs are not immune to it.

The second finding concerns individual profitable runs. They depend on concentrated returns: one or two lucky outcomes make the whole series positive. If you look only at the best runs, it is easy to mistake luck for skill. That is why the benchmark builds in paired tests and a correction for multiple comparisons, and why average profit is computed across all configurations.

The numbers add up as follows:

• 28 "model plus scaffold" configurations;
• 324 races;
• 7 significant improvements from revision and 2 significant degradations;
• 0 configurations with positive average profit under capped-Kelly;
• 4 models in a separate comparison of the two workflows, where the ready-made program mode wins.

Capped-Kelly sets bet size using the Kelly criterion with a ceiling. It penalizes overestimating your own probabilities, and that is exactly what exposes the agents' weakness.

The practical takeaway for anyone building forecasting agents, whether for demand, prices, risks or market events, is this. Evaluate four things separately: predictive competence, ability to adapt, resource spending, and the value of the decision itself. An improvement in a forecast metric does not guarantee a positive return, and a cheap reusable program can beat expensive per-case research. The code and project page are published on GitHub and on the HKUST KnowComp website, so the protocol can be carried over to other domains with repeated events.