Yandex Grant Funds an LLM Engine: 35B Model on RTX 2080 Ti Runs Three Times Faster Than llama.cpp
The author spent a 600,000-ruble Yandex grant on TinyCoder, a from-scratch C++ and CUDA engine for running neural networks at home. The Ornith-1.5-35B-A3B model weighs 34.4 GB, while the older RTX 2080 Ti has only 11 GB, so part of the model is loaded on the fly. Even so, it generates 53.6 tokens per second versus 17.1 for llama.cpp, roughly three times faster. The gap nearly disappears on large dense models, so test on your own workload.
AI-processed from Habr AI; edited by Hamidun News
Running large neural networks at home is constrained not by GPU compute but by memory: most developers have a card with 8–12 GB, while interesting models weigh tens of gigabytes. A Russian developer received a 600,000-ruble grant from Yandex and spent it on TinyCoder, an engine for local models written from scratch in C++ and CUDA. The headline number: the Ornith-1.
5-35B-A3B model, 34.4 GB in size, runs on an RTX 2080 Ti with 11 GB of memory and generates 53.6 tokens per second.
A token is roughly a word or part of one, so this is a measure of how fast the answer is typed out. The popular llama.cpp engine delivers 17.
1 tokens per second in the same scenario, which is three times slower.
Where the money came from and why an engine
At the start of 2026, Yandex launched grants for open-source projects on the SourceCraft platform. The author submitted np, a C++ analogue of NumPy with CUDA support and hand-written SIMD optimizations (AVX2, AVX-512, AMX). According to his measurements, it is 2–20 times faster than the original and uses one and a half times less memory. The project was among the winners, and 600,000 rubles arrived in the account.
Next came the question of how to spend it. Rewriting all of NumPy made no sense, since the author had already solved the acceleration task. He chose a local AI coding agent and then focused on the engine underneath it. He considered a cloud VM with a GPU too expensive and worked on a home PC: an Intel i7-4790K, 32 GB of DDR3, and an RTX 2080 Ti with 11 GB.
Most of the engine's code was written by neural networks. The author first used DeepSeek V4, then DeepSeek V4.1 Flash; in his words, they cost less and produced better code than gpt-oss-120b or Qwen3-235B. On October 11, 2026, SourceCraft went down after an incident at a Yandex data center, and about 200,000 rubles remained in the grant account. Yandex's cloud models became unavailable, and he switched to an external DeepSeek provider. For a project about a local agent, this is a useful argument: it should not depend on other people's services.
How a 34 GB model fits into 11 GB
tinycoder-inference is not a wrapper around ggml or PyTorch. Its only external dependencies are the np library and cuBLAS for dense matrix multiplication. The dequantization kernels, int8 matrix multiplication, flash attention, and samplers were written by the author. The engine reads GGUF formats (quantizations from Q2_K to Q6_K, the IQ series, and the ternary pq2_0) and EXL3 directly, without conversion and without Python on every token.
The main trick concerns Mixture-of-Experts (MoE) models. Such a model consists of many "experts," but only a small subset is active for each token. So the dense part stays in video memory, while the experts are streamed in through an arena layer. Picture a library with a small reading room: the books you need are on the desk, and the rest are fetched from storage on request.
The context planner also works on the fly: before loading, it measures free VRAM and RAM and selects the KV-cache precision itself. On an RTX 2080 Ti, this yields up to 196,352 tokens of context for a 9B model and up to 64,256 for a 27B model.
What the measurements showed and where the limits are
Measurements follow the pp512/tg128 scheme: processing a 512-token prompt and generating 128 tokens, in three pairs of alternating runs.
- Qwen2.5-Coder-1.5B q2_k: prompt processing 12,240 versus 7,679 tokens/s for llama.cpp; generation 325.9 versus 321.6.
- The same model in iq3_xxs: 12,243 versus 10,007; generation 273.8 versus 262.2.
- Ornith-1.5-35B-A3B in hybrid mode: prompt 176.3 versus 132.2; generation 53.6 versus 17.1.
- The ternary Bonsai-27B: 880.6 versus 878.5 on the prompt and 51.7 versus 50.4 on generation, which is near parity.
- Qwen3.8-27B in EXL3 against exllamav3: prompt 727.5 versus 719.5, while generation is slightly slower at 23.60 versus 23.76.
These numbers should be read soberly. They are one author's measurements on one Turing-generation card. Ampere (RTX 3090) is still under debugging, and newer cards run via compatible PTX JIT or base SASS. The clear gain appears on light quantizations and MoE offloading, while on large dense models the engine has roughly caught up with its competitors.
Gateway and plans
To make the engine work with any agent, the author wrote the tinycoder-openai gateway in Go. It supports /v1/chat/completions, /v1/responses, and the Anthropic Messages API, as well as tool_calls and SSE streaming. Models load on the first request and are unloaded from VRAM after a timeout, and generation can be interrupted inside the layer loop. The executable is 9 MB and the native library is 145 MB. Near-term plans include speculative decoding and kernels for modern GPUs.
The practical takeaway: if you run local models on a consumer card, especially a large MoE model in limited VRAM, it is worth running your own workload on TinyCoder and comparing it with llama.cpp. The project is young, but its results on older hardware suggest there is still speed to be found.
- Nov 22, 2026Nvidia Shares Fell 14% on Friday and 46% Over Seven Weeks, Erasing $2.6 Trillion
- Nov 22, 2026SEC Probes Nvidia's Accounting as Shares Drop 14% in One Day
- Nov 22, 2026Nasdaq 100 Falls 10.4% in a Week, Its Worst Stretch Since the Pandemic
- Nov 22, 2026LangChain, a Side Project Turned $3 Billion Startup in Four Years, Raises $300 Million
Key Questions Answered
Where the money came from and why an engine?
At the start of 2026, Yandex launched grants for open-source projects on the SourceCraft platform. The author submitted np, a C++ analogue of NumPy with CUDA support and hand-written SIMD optimizations (AVX2, AVX-512, AMX). According to his measurements, it is 2–20 times faster than the original and uses one and a half times less memory. The project was among the winners, and 600,000 rubles arrived in the account. Next came the question of how to spend it.
How a 34 GB model fits into 11 GB?
tinycoder-inference is not a wrapper around ggml or PyTorch. Its only external dependencies are the np library and cuBLAS for dense matrix multiplication. The dequantization kernels, int8 matrix multiplication, flash attention, and samplers were written by the author. The engine reads GGUF formats (quantizations from Q2_K to Q6_K, the IQ series, and the ternary pq2_0) and EXL3 directly, without conversion and without Python on every token. The main trick concerns Mixture-of-Experts (MoE) models.
What the measurements showed and where the limits are?
Measurements follow the pp512/tg128 scheme: processing a 512-token prompt and generating 128 tokens, in three pairs of alternating runs. Qwen2.5-Coder-1.5B q2_k: prompt processing 12,240 versus 7,679 tokens/s for llama.cpp; generation 325.9 versus 321.6. The same model in iq3_xxs: 12,243 versus 10,007; generation 273.8 versus 262.2. Ornith-1.5-35B-A3B in hybrid mode: prompt 176.3 versus 132.2; generation 53.6 versus 17.1.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.