At RGS, local LLMs cut the review of 500 tasks from five working days to two

Before every planning cycle at RGS, development teams bring in hundreds of tasks, and each one has to be scored against several prioritization criteria. One of them is IT compliance: does implementing the task help the company meet legal requirements? Anastasia, who heads the IT compliance function, described on Habr how she cut the review of about 500 tasks from five working days to two. She had no development budget and cannot program. The whole pipeline runs on free tools on a laptop.

Why a keyword filter didn't work

The pilot involved two "trains," meaning two large development units. The first cycle alone produced about 500 tasks, and reviewing them took a week of the only expert's time. For that whole week, the work compliance exists for stood still: analyzing new requirements and supporting complex projects. Onboarding the remaining trains would have eaten most of a quarter, and expanding the team to cover peak load was not an option.

Keyword search did not work: teams word things differently, and what needs scoring is meaning. So Anastasia decided to test the hypothesis as cheaply as possible. She made an anonymized export of tasks and gave it to several external models along with instructions on the criteria. After a few prompt iterations the result stabilized, but agreement with expert ratings was only about 60%, and the errors mostly leaned toward overestimation. That was enough to start with: an overcautious model reliably filters out what the expert doesn't need to look at.

A local build: n8n,

Ollama, and a database of verified ratings

The first version had three weak points: manual preparation of exports, loss of meaning during anonymization, and dependence on external services that may stop working in Russia. A corporate solution didn't work out right away, so Anastasia chose a local model. The data never leaves the computer, and no external services are needed. The price is the laptop's limits: tasks are fed in batches, and models that were too heavy froze the machine completely.

The orchestrator is n8n, a visual workflow builder. She ran models through LM Studio and Ollama and ultimately settled on Ollama. The pipeline works like this: export the tasks, strip technical junk from the text, pass the text to the model together with the instructions, get an answer in a strict format (a level on the scale plus a short justification), and save the result.

The key component is a database of already verified ratings used with RAG. Given a task, the model finds similar ones that the expert has rated and relies on them. Without this database, the first builds showed only 20% accuracy. After refining example selection, the prompt, and the set of models (Qwen and GPT-oss), the build reached 70–80% on the test set and processed 500 tasks in about 12 hours. That is roughly 1.5 minutes per task.

To compare variants fairly, Anastasia used AI to write an ordinary program. It matches the ratings from several algorithms against the expert's and computes four metrics: exact hits, hits within one step of the scale, filtering out unnecessary items, and the direction of errors. Overestimation means an unnecessary review; underestimation means a risk of missing an important task.

From 55–60% on the live stream to 78–82%

The test set turned out to be optimistic. On real tasks, the first good build managed only 55–60%. The live stream contains more new wording and new systems, and some tasks come from trains that are only now joining prioritization. Anastasia then tried several ratings from different models, or from one model with different parameters, plus additional knowledge and enriched context, including via MCP. The selection criterion was simple: if a variant's accuracy dropped when new trains were onboarded, it was discarded.

The surviving builds run longer than the initial one, 12–24 hours for 500 tasks, but they deliver 78–82% on real data, and overestimation and underestimation errors are now balanced. Instead of a filter for the expert, the result is an algorithm that fills in the values itself, and the expert checks them as needed. The economics are straightforward: computer time is cheaper than expert time, and the more accurate the algorithm, the less manual checking is needed.

The scope of the process changed too. Previously only new tasks were scored by hand, and modified ones were not rechecked for lack of time. Now the algorithm itself exports new tasks and tasks with changed descriptions before implementation starts, assigns ratings, hands over a file for review, and writes the criterion value into the field used to calculate priority. It can write the value immediately or after review. The process now serves nine trains instead of two.

Limitations and what's next

The solution has weak points, and the author names them. It is tied to one laptop and one expert, and 80% accuracy is good for a local model but does not eliminate spot checks. Anastasia plans to recheck periodically whether accuracy is declining. A corporate instance of the algorithm already runs in a limited mode and delivers comparable accuracy six times faster. By Anastasia's assessment, what remains is lifting the limitations.

For practitioners, the takeaway is this. The "read, match against rules, score" pattern fits many operational processes. The working minimum for it is an accumulated database of expert ratings for RAG, metrics that account for the direction of errors, and testing on the live stream, not just on a sample. RGS is already launching a similar process in test mode. The expert's role shifts as a result: the expert manages the process and makes decisions where they are truly needed.