Trader Turns RAG and RandomForest into BUY/SELL Signals: 62% Accuracy on 59 Records

Many people can bolt an LLM onto a news feed and get a "smart analyst." Far fewer honestly show what happens next: when the goal is not to retell the news but to predict price. The author of a Habr project continued his RAG experiment and took it all the way to trading signals with explanations. Here is what is worth borrowing from this story.

One Pipeline Instead of Three Competing Tasks

The first version of the system ran three background tasks in parallel: classification, entity extraction, and enrichment. Each pulled its own model into GPU memory, and Ollama constantly unloaded and reloaded weights. When the database grew from 12,000 to 37,000 articles, this became too expensive.

The solution is simple: a single pipeline. An article goes through classification, junk filtering, market-impact scoring, NER, summarization, and embedding. If the filter decides a news item is about sports or culture, it is marked as skipped, and not a second more of GPU time is spent on it. The lesson is universal: cheap filtering first, expensive model second.

The Ollama settings were also tuned by hand. The author allowed two models to stay in memory (Qwen 14B and BGE-M3), set parallelism to 6, and pinned everything to a single Tesla P100. At a value of 8, memory hit its ceiling and requests failed with timeouts. The second card is deliberately not used: when layers spread across GPUs, speed drops 3–5x because of data transfer over PCIe.

From News to Signal

Example: NER finds "Sber" in the text. A synonym dictionary, replenished from the Moscow Exchange API, turns it into the ticker SBER. For each day and ticker, features are computed: the sum of news directions (+1 for positive, −1 for negative), the number of news items, 5- and 20-day volatility, and average trading volume. Then a RandomForest with Platt scaling calibration outputs BUY, SELL, or NEUTRAL and a probability, and the API also returns the list of news items that influenced the decision. In Streamlit this becomes a terminal with candlesticks and expandable evidence.

An amusing detail: for Alfa-Bank, which is not traded on the exchange, a dummy ticker ALFAMOCK was created. News accumulates in advance, and after an IPO the ticker will simply be replaced.

Where Everything Breaks

The most valuable part of the article is the analysis of failures. An attempt to add time decay made the feature fractional (0.85 instead of 1), and accuracy fell from 52% to 36%. On 55 records this is just noise, so the idea was postponed until 200+ records.

Another case: VKCO received BUY with 64% confidence amid neutral news. The model simply learned that the market rises in 55% of cases. The author rejected a hard "no pressure means NEUTRAL" filter as masking the symptom and planned dominance features, but also only after more data is collected. The technical debt is recorded and documented.

Validation uses the walk-forward method: the training window slides through time so as not to hit a "lucky" period by chance. The target scheme is 30 days of training and 7 of testing, but for now a trimmed-down 14/3/3 setup is used. The result is an average accuracy of 62% (median 67%) on 45 test examples.

What to Do With This

Do not treat 62% as a trading strategy: the sample is tiny, commissions and slippage are not accounted for, and the model can easily learn noise. The project's value lies elsewhere. It is a good architecture template: an early filter to save GPU, explainable output with sources, calibrated probabilities, and honest recording of decisions in ADRs. If you are building a similar pipeline on local models, it is worth adopting the Ollama tuning approach and the habit of measuring the effect of each feature before keeping it. The code of the public part is open on GitHub.