AlphaBet trains an LLM agent to forecast events: Brier score improves 14.4%

Researchers from Liqin Ye's team (the paper lists the Georgia Institute of Technology) have released AlphaBet, an agentic reinforcement learning framework that turns a language model into a forecaster of real-world events. After training, the base GLM-4.5-Air model cut its Brier score on the Forecast-Dojo benchmark from 0.731 to 0.626, and on FutureX from 0.612 to 0.528. In relative terms, that is 14.4% and 13.7%. A lower Brier score means the probabilities match real outcomes more closely. The code, models, and data are openly available.

Why you can't train forecasting on a ready-made set of facts

Forecasting an event that hasn't happened yet involves two tasks: finding relevant evidence and deciding how strongly it supports each outcome. Many outcome-based training schemes for LLM forecasters start from materials collected in advance for a single prediction date. That leaves evidence search outside the training loop: the model learns to weigh what it is given, but not to find what it needs.

AlphaBet builds search directly into the policy. A single model searches for information, reasons, and outputs probabilities at several dates before the event resolves. Each dated forecast is scored against the final outcome with a proper scoring rule. One resolved question thus becomes a source of signal for several information states at once: a month before the event, a week before, a day before.

Two gaps in forecast error

The authors split a forecaster's regret into two components:

• evidence-acquisition gap: the model did not find or request the necessary data;
• probability-reporting gap: the data was found, but the output number is poorly calibrated.

Controlled ablations show that the gain comes from multi-date training and combines improvements in both parts. RL teaches the model to search better and to translate what it finds into percentages more accurately.

The catch with normalization in GRPO

A separate result concerns Group Relative Policy Optimization. The standard implementation normalizes the advantage within a group of samples by the standard deviation of the reward. The authors show that with multi-class proper scoring rules, this normalization can distort the target distribution the model is supposed to reproduce when it reports probabilities.

The empirical results back this up. Normalizing by std degrades the proper-scoring-rule metric, while the accuracy of guessing the most likely outcome barely changes. The model keeps naming the right favorite but assigns the percentages less well. For tasks where calibration matters more than a yes/no guess, this is a significant caveat to the usual recipe.

Key details of the work

• Framework: AlphaBet, agentic RL with a single policy for search, reasoning, and probability reporting.
• Base model: GLM-4.5-Air.
• Forecast-Dojo: Brier 0.731 → 0.626, a 14.4% improvement.
• FutureX: Brier 0.612 → 0.528, a 13.7% improvement.
• Key technique: training on several dates before the event resolves, with each one scored by a proper scoring rule.
• Theoretical framing: regret decomposed into an evidence gap and a probability gap.
• GRPO finding: normalization by standard deviation degrades proper scoring and barely affects top-outcome accuracy.
• Availability: code on GitHub (AlphaBet-Agent/AlphaBet), models and data in the liqinye/alphabet collection on Hugging Face, project site alphabet-agent.github.io.

The paper's authors are Liqin Ye, Rongzhi Zhang, Xin Liu, Zheng Li, Yuan He, Qingyu Yin, Priyanka Nigam, and Chao Zhang.

What practitioners should take from this

Teams building agents for analytics, risk, or market expectations should revisit their metrics. If you measure only the share of correct answers, you cannot see calibration, and calibration is what determines a forecast's value in decision-making. Brier or another proper scoring rule will show the difference that accuracy hides.

The second takeaway concerns training. If you train an agent with GRPO on tasks with probabilistic output, check how std normalization affects calibration and compare variants without it. The third: a single event resolution yields several training examples at different dates. That is a cheap way to extract more signal from a limited set of resolved questions.

The open weights and data make it possible to reproduce the result and compare it with related work, for example Forecast-Dojo, with its reproducible environments for training forecasters, and research on jointly training action and prediction in language agents. So far, the numbers come from one base model and two benchmarks, so transfer to other LLMs and domains remains a question for independent verification.