T-Bank improves AI Code Completion with a filter that suppresses unhelpful suggestions

T-Bank described how it reworked its internal AI Code Completion for 7,500 developers and raised the share of accepted suggestions without changing the core generation model. Instead of another attempt to make completion "smarter", the team added a separate filter that decides whether to show a suggestion at all.

Where the ceiling was

T-Bank's code autocompletion service has run in production for several years and is used daily by almost all internal developers, about 7,500 unique users. The core metric, Acceptance Rate, hovered around 20% for a long time: roughly one in five suggestions was accepted, and the rest were ignored.

The team tried lengthening suggestions, changing the generation strategy and expanding the number of places where suggestions appear, but this only produced more noise. The more actively the system suggested, the more often developers saw useless continuations, and the weaker their trust in the product became.

That led to a different hypothesis: the problem might lie not only in generation quality, but also in the lack of a separate decision about when a suggestion should be hidden.

The team noticed an important behavioral effect: when there is less junk, users press Tab more willingly and more often give a chance even to non-obvious suggestions. By internal estimates, each additional percentage point of AR added about 2% to the number of acceptances over time. But there was a hard business limit: the filter was not allowed to instantly "eat" more than 5% of suggestions that were already being accepted.

What the filter is made of

The first step was a quick CatBoost baseline. The model was trained as a binary classifier: accept the suggestion or not. It used only features that can be computed in real time without storing request history: IDE, programming language, cursor position, suggestion type, and prefix and suffix size. Even this simple version gave about +2.3 pp to Acceptance Rate offline and confirmed that the task contains a strong signal.

Next, the team moved to a text filter based on Qwen2.5-Coder 1.5B. Larger models did not fit production constraints: a target of 30 requests per second on a single Nvidia A100 and p90 latency of no more than 50 ms. So the team chose a compromise: compact enough for inference, yet still specialized for code. To keep the model from confusing file context with the suggestion itself, the input had to be strictly structured and the model fine-tuned for classification rather than generation.

• The generation head was replaced with binary classification
• Context was marked up with tags for prefix, line, answer and suffix
• IDE, language and cursor position were encoded as special tokens
• The final stage added fine-tuning, LoRA and focal loss to handle class imbalance

This pipeline improved quality step by step: strict structuring raised the gain to about +3.9 pp, special tokens to +5.1 pp, and full fine-tuning brought the offline result to +6.8 pp. What mattered most was not only the fine-tuning but how the input was packaged: the model got better at telling file context apart from the suggestion it needed to evaluate, not rewrite.

What broke in production

Everything looked great on synthetic tests, but a shadow launch quickly cooled expectations. Simply converting the model to ONNX nearly tripled throughput and cut response time to about 30 ms, yet on real traffic latency at peaks shot back up to 90 ms. The cause was not the model itself but the load profile: production received bursts of nearly simultaneous requests that the tests did not include. The team fixed it with Triton and dynamic batching, using a small batch size and a short queue wait.

"Offline is necessary, but shadow mode is the only place where reality begins."

Then a second layer of problems surfaced: the filter was too aggressive. To keep the loss of accepted suggestions within 5%, the threshold had to be retrained on a weekly window of data instead of a few days. The team then added another CatBoost on top of the LLM, fed with the main model's score, tabular features and historical signals such as the interval between requests and changes in prefix length. User state was stored in Redis for this.

Along the way the team hit a classic engineering bug: some features in production were computed in bytes and others in characters. After aligning the logic, the A/B test showed 4.7% of traffic dropped and +5.2 pp of Acceptance Rate, with no skew across languages or IDEs.

What this means

T-Bank's case shows that the next quality gain in AI tools does not always come from a new, larger model. Sometimes a separate decision layer that knows when to stay silent has a greater effect. For high-frequency products, it is also a matter of trust: when unnecessary suggestions are removed, users are not only less annoyed but, over time, accept useful options more often. At the scale of thousands of developers, this quickly turns into noticeable time savings.