Quant-Arena: None of 9 LLMs Beat the A-Share Market in 2024

Nine frontier models tried their hand at being quant researchers on real data from China's A-share stock market. None of them beat a simple equal-weighted market portfolio in 2024. GPT-6 Astra posted the best result on the leaderboard, but even that did not translate into excess returns on the held-out data. This comes from Quant-Arena, a new benchmark and public leaderboard for end-to-end quant research, described in a paper on Hugging Face Daily Papers.

The Full Cycle Instead of Isolated Pieces

Quant work is a loop. A researcher states a market hypothesis, turns it into an executable signal, runs it against historical data, and decides what to keep and what to discard. Existing benchmarks test individual links in that chain: the correctness of a formula, the quality of code, or a single trading decision. The open question is whether an LLM can run the whole loop and produce a strategy that works outside the period it was designed on.

Quant-Arena asks exactly that. The model builds its own features and writes the stock-scoring code. It receives feedback only on 2022–2023. It then freezes a single strategy, which is tested on a sealed 2024 that the model has never seen.

Cash Account, Fees, and the Order Book

The most important part of the benchmark's design is the evaluator. It operates as a cash account: it applies trading constraints and costs, and it estimates market impact using a simulated order book. This closes a typical hole in many backtests, where a pretty return curve comes from assuming that any volume can be bought at the current price with no consequences.

The final score has three components:

• realized excess return, the main component, which carries most of the weight;
• signal quality;
• diversity of the features the strategy uses.

The third item guards against degenerate solutions. A model that finds one working factor and replicates it in ten variations will score lower than one that assembles genuinely different sources of signal.

The Best Result Is Still Below the Market

The authors tested nine frontier models. GPT-6 Astra took first place on the overall score. But that score includes signal quality and diversity, while the decisive test is money on the sealed year. Here the picture is harsh: no model beat the equal-weighted market in 2024.

A cautious conclusion follows. LLM agents can write code, generate features, and improve metrics on the period available to them. That does not yet guarantee durable out-of-sample returns. One possible explanation is overfitting to 2022–2023: the agent gets feedback only on those two years and optimizes for them. A shift in the market regime may also play a role. But these mechanisms remain hypotheses, and there is only one confirmed fact: an idea that looked good on historical data did not survive the sealed year.

The equal-weighted market is not an arbitrary bar. It is a boring, cheap, and hard-to-beat benchmark, and every active manager is compared against it. If an agent cannot beat it after accounting for costs and price impact, it is too early to talk about automating alpha research.

How to Use Quant-Arena

The benchmark is open to submissions from any model or agent harness. That is a key detail: the leaderboard can compare not only base LLMs but entire systems with memory, tools, self-checking loops, and disciplined hypothesis selection.

For teams building agents for finance, the value is practical. First, it is a ready-made testbed with a strict separation of periods, which makes it easy to catch overfitting to history. Second, the feature diversity metric shows whether an agent is exploring the hypothesis space or just spinning its wheels. Third, accounting for market impact gives a more honest estimate of how well a strategy scales than a backtest without costs.

For investors and portfolio managers, the takeaway is simpler: loud claims about autonomous AI traders should be checked against out-of-sample data. As long as the best model in Quant-Arena stays below a passive portfolio, the role of LLMs on quant teams looks like an assistant for code and ideas, not a replacement for the researcher.