Bitrix24 describes what it is building around its own 1-trillion-parameter LLM

At Saint HighLoad++, Bitrix24 described what begins after you say "we deployed our own LLM" with 1 trillion parameters. The weights of such a model alone take about 1 TB even in FP8, which is almost an entire eight-GPU server. Chats, long-document processing and agents all run on that same hardware at once. The Habr announcement only lays out the problem; the details are in the full article. Below is a breakdown of the engineering questions behind it.

Where "simple" ends

The arithmetic of the project is unforgiving. One FP8 parameter weighs one byte, so a model with 1 trillion parameters needs about 1 TB of memory before the first request arrives. In BF16 it comes to about 2 TB, and in 4-bit form about 0.5 TB. Eight H200s provide 1128 GB of HBM, so FP8 weights almost fit on a single node, but very little is left for the KV-cache. The platform therefore almost inevitably has to rely on parallelism across several nodes and on saving memory.

The model itself is not the hard part: open weights at this scale exist, and they are usually MoE architectures, where only a fraction of the parameters is active per token. The real work starts around the model, and that is what the talk is about.

Three workloads on the same GPUs

The author starts from the premise that three different types of traffic share the compute, each with its own requirements:

• Chats. Short requests, with a user watching the screen. Time to first token and a steady generation speed matter most.
• Long documents. A heavy prefill stage that can occupy a GPU for a long time and stall its neighbors.
• Agents. Chains of dozens of calls, often with a shared prefix. Throughput is critical, while the latency of any single call matters less.

This leads to the three questions from the announcement: who waits, what fills the resources between peaks, and how to check quality after an optimization. The first is solved with priorities and queues. A chat should not sit behind the processing of a multi-thousand-token page, while a background agent can easily tolerate a pause. The second is solved by filling idle time. At night and in the troughs between peaks, you can run batch jobs that get preempted when interactive traffic arrives.

Optimizations and the risk of silent degradation

The third question is the most unpleasant one. Quantization, prefix caching, speculative decoding and changes to batching speed the system up, but they can quietly make answers worse. The error does not crash the service and does not show up in latency metrics. The model simply confuses facts a little more often or holds the tool-call format a little worse. For agents this is critical: a single broken JSON structure breaks the whole chain.

The only way to catch this is to run a fixed set of tasks before and after every change and compare the results. The set must include real product scenarios: long context, function calling, the Russian language. General benchmarks are almost useless here.

What to check before launch

Anyone considering their own large model should go through a short list:

• Calculate memory: weights, KV-cache at the real context length, and headroom for peaks.
• Split traffic into classes and agree in advance which class can be delayed.
• Decide what will fill idle time and how those jobs get interrupted.
• Build your own quality evaluation set before the first optimization, not after.
• Measure not only average speed but also latency tails: p95 and p99 for chats, throughput for agents.

The talk's main point is that your own LLM with 1 trillion parameters is a platform with a scheduler, queues and a quality control system, not a single model on a server. Bitrix24 discloses the specific load figures and the cluster configuration in the full version of the material on Habr.