An Engineer Ran a 125-Billion-Parameter Model on a Home PC at 22–25 Tokens per Second
Banks and state corporations don't allow code and tasks to go to cloud AI services, so the author set up a 125-billion-parameter model under a desk: a 16 GB GPU and 128 GB of regular system memory. It's a MoE model: only about 6 billion parameters are active for each token (a piece of a word). The Strata engine delivers 22–25 tokens per second, and the compressed version matches the 8-bit reference in 93.1% of cases. For large-scale agentic tasks with closed data, a local setup already works.
AI-processed from Habr AI; edited by Hamidun News
Imagine your AI agents write code, analyze incidents from logs, and maintain the backlog, while the client's security team asks where task descriptions, code snippets, and data are going. If the answer is "to someone else's servers in the cloud," the conversation often ends there. The author of this review works with state corporations, banks, and industrial holdings, and solved the problem radically: they set up a 125-billion-parameter model under a desk workstation (this is how neural network size is measured; "125B" in the title refers to exactly this).
It runs on a GPU with 16 GB of memory and produces 22–25 tokens per second. A token is a piece of a word, so the figure means the speed at which the model "types" its response. Not a single byte of data leaves the company's perimeter.
Why the cloud and masking don't close the question
Agent workflows send more to cloud APIs than people expect: project names, counterparties, amounts, internal system URLs, database schemas, and sometimes access keys. Add metadata: who asked, when, and how often. On top of that come the requirements of Russia's Federal Law 152-FZ, data retention policies that can change with a single announcement, limits on parallel sub-agents, and bills: one investigative agent can consume millions of input tokens in a single evening.
Masking (replacing names, tax IDs, and amounts with placeholders) helps, but only partially. A contract and its client can still be guessed from the JSON structure and task identifiers. If you replace the industry with ORG_1 and the product with PRODUCT_2, the model loses the domain context and starts to get confused. Code and configs can't be cleaned with regular expressions, metadata remains, and the masking pipeline itself has to be maintained. The author's conclusion: tasks involving code and internal data should be handled only locally.
How a 125B model fits into 16 GB of VRAM
The author tried models in the 30–35B class: they weren't sufficient for long context, strict response formatting, and tool calling. A regular ("dense") 125B model on home hardware would produce single-digit tokens per minute, because every token requires passing through all 65 GB of weights, even at 4-bit precision. The MoE (mixture-of-experts) architecture saved the day: each token activates only about 6 billion parameters, while the remaining experts sit in system RAM and are loaded as needed. Another roughly 51 billion parameters belong to n-gram tables, which are read from RAM line by line. VRAM holds the attention layer, the shared layers, and the "hot" experts.
The hardware is ordinary: an RTX 5060 Ti with 16 GB, an older 8-core Xeon E5-2640 v3, and 128 GiB of DDR4. The main bottleneck is memory bandwidth: about 25 GB/s on random expert reads. The start was weak: around 3 tokens per second in LM Studio, and 9.2 after manually tuning llama.cpp. The Strata engine (MIT license), written specifically for this architecture, delivered about 437 tokens/s for prompt processing and 25–28 tokens/s for generation with the IQ3_S quant after calibration for the hardware. That is roughly three times faster than the best llama.cpp result. The final 22–25 tokens/s came with a higher-precision quant and a native context of 262,144 tokens.
What helped and what didn't
- Engine version. An update to Strata, together with a speculative decoder (which "guesses" several tokens ahead), gave +79% to generation speed. The author recommends recalibrating the parameters after every update.
- Quantization. The choice affects not quality but how VRAM is laid out. The main variant matches the 8-bit reference in 93.1% of cases.
- Pagefile on Windows. The limiting factor was not physical memory but the commit limit (RAM plus the swap file). After increasing the pagefile, generation rose from 15.1 to 22.3 tokens/s (+48%).
- Context and slots. Long context is almost free, but parallel slots are expensive: a second slot takes VRAM away from the expert cache and roughly halves the speed per request.
Limitations and practical takeaway
This is personal experience on specific hardware, not an independent benchmark. The numbers depend on memory speed, engine version, and prompt length, so the author compares configurations only on identical prompts of about 10,000 tokens. Running several agents simultaneously on such a workstation is limited.
The practical takeaway for teams: if you have sensitive data or strict information security requirements, don't start with masking. Evaluate a local setup with a MoE model first. Budget time for calibration and don't skimp on RAM. Preparation also matters: skills with well-tuned processes, pre-built fact bases, and verifiable results (tests, gates, end-to-end checks) matter more than generation speed. A local setup also serves as a backup channel in case subscriptions lapse or the internet goes down. The author's forecast: a hybrid approach, where the cloud handles design and complex decisions while a local model takes on bulk task waves and everything sensitive, will become the norm for development teams within the next one to two years.
- Nov 21, 2026How to Keep an AI Agent Within Its Task: Code Checks the Plan Before Any Site Edits
- Nov 21, 2026Modal: AI agent sandboxes use less than 6% of peak CPU for over half their lifetime
- Nov 21, 2026Two Tools Instead of 2,500 API Methods: How to Keep an Agent from Burning 1.17 Million Tokens of Context
- Nov 21, 2026AGENTS.md helped Codex follow project rules: 0 violations out of 13 versus 5 out of 13
Key Questions Answered
Why the cloud and masking don't close the question?
Agent workflows send more to cloud APIs than people expect: project names, counterparties, amounts, internal system URLs, database schemas, and sometimes access keys. Add metadata: who asked, when, and how often. On top of that come the requirements of Russia's Federal Law 152-FZ, data retention policies that can change with a single announcement, limits on parallel sub-agents, and bills: one investigative agent can consume millions of input tokens in a single evening.
How a 125B model fits into 16 GB of VRAM?
The author tried models in the 30–35B class: they weren't sufficient for long context, strict response formatting, and tool calling. A regular ("dense") 125B model on home hardware would produce single-digit tokens per minute, because every token requires passing through all 65 GB of weights, even at 4-bit precision. The MoE (mixture-of-experts) architecture saved the day: each token activates only about 6 billion parameters, while the remaining experts sit in system RAM and are loaded as needed.
What helped and what didn't?
Engine version. An update to Strata, together with a speculative decoder (which "guesses" several tokens ahead), gave +79% to generation speed. The author recommends recalibrating the parameters after every update. Quantization. The choice affects not quality but how VRAM is laid out. The main variant matches the 8-bit reference in 93.1% of cases. Pagefile on Windows. The limiting factor was not physical memory but the commit limit (RAM plus the swap file).
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.