FINESSE-Bench: Evaluating LLMs in Finance Without Fooling Yourself
The AI lab at Finam, a Russian fintech company, has published an updated version of FINESSE-Bench, a test suite that measures the financial knowledge of language models. The problem it targets: LLMs often score high on standard general-purpose benchmarks but cannot solve real financial tasks such as currency forecasts, risk analysis and trading strategies. FINESSE-Bench contains two types of tests: CFA-like Level 1 (financial exams) and CFTe-like Level 1 (technical analysis). The methodology has been strengthened with bootstrap estimates and an analysis of how well knowledge transfers between task types.
AI-processed from Habr AI; edited by Hamidun News
The AI lab at Finam has published an updated version of FINESSE-Bench, its open benchmark for measuring how well LLMs work in the financial domain. It is not just an expansion of the previous version: the methodology has been reworked and the coverage of financial tasks widened.
Why a dedicated financial benchmark is needed
The core problem: LLMs that post excellent results on popular open benchmarks (MMLU, ARC, HellaSwag) often fail at real financial tasks. In practice, Finam observes a systematic gap between overall accuracy and accuracy on financial scenarios.
Finance requires correctly predicting risks, understanding currency pairs, interpreting financial reports and analyzing price charts. A mistake in the financial domain can cost millions.
For this reason, FINESSE-Bench does not simply test knowledge. It evaluates how a model behaves when:
- Task difficulty increases
- Quality carries over between types of financial tasks
- It faces specialized scenarios (trading, risk management, analytics)
What changed in the new version
Quite a lot has changed since the first version of FINESSE-Bench:
- Expanded datasets: new technical analysis tests have been added (CFTe-like Level 1)
- Updated CFA-like: problematic questions in the financial exams block have been fixed
- Wider model pool: testing covers more LLMs
- Improved metrics: a bootstrap estimate has been added to reduce variance in the results
- Transfer analysis: a check of how far quality carries over between types of financial tasks
- Saturation analysis: an assessment of how well the question sets themselves differentiate between models
Why this matters for financial AI
Fintech companies are constantly experimenting with LLMs for:
- Analyzing financial news and its effect on prices
- Building portfolios automatically
- Checking compliance with regulatory requirements
- Generating investment recommendations
- Trading algorithms that react to text
A well-calibrated benchmark prevents costly mistakes, where a model looks smart but loses capital in practice.
What it means
The rise of domain-specific benchmarks (financial, medical, legal) shows the AI industry maturing. LLMs can no longer be judged on general tests alone. Every sector needs its own measuring stick. FINESSE-Bench is an example of how companies deploying AI are forced to build their own checkpoints to make sure a model can really work in their domain, rather than merely having a high score on a benchmark from the internet.
Key Questions Answered
Why a dedicated financial benchmark is needed?
The core problem: LLMs that post excellent results on popular open benchmarks (MMLU, ARC, HellaSwag) often fail at real financial tasks. In practice, Finam observes a systematic gap between overall accuracy and accuracy on financial scenarios. Finance requires correctly predicting risks, understanding currency pairs, interpreting financial reports and analyzing price charts. A mistake in the financial domain can cost millions. For this reason, FINESSE-Bench does not simply test knowledge. It evaluates how a model behaves when: Task difficulty increases Quality carries over
What changed in the new version?
Quite a lot has changed since the first version of FINESSE-Bench: Expanded datasets: new technical analysis tests have been added (CFTe-like Level 1) Updated CFA-like: problematic questions in the financial exams block have been fixed Wider model pool: testing covers more LLMs Improved metrics: a bootstrap estimate has been added to reduce variance in the results Transfer analysis: a check of how far quality carries over between types of financial tasks Saturation analysis: an assessment of how well the question sets themselves differentiate between models
Why this matters for financial AI?
Fintech companies are constantly experimenting with LLMs for: Analyzing financial news and its effect on prices Building portfolios automatically Checking compliance with regulatory requirements Generating investment recommendations Trading algorithms that react to text A well-calibrated benchmark prevents costly mistakes, where a model looks smart but loses capital in practice.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.