Inkling 2 Mini packs a million-token context into 16 GiB instead of 96
When a neural network reads a long text, it stores notes for every token (a piece of a word) in the KV cache, and at a million tokens that cache swells to tens of gigabytes. Inkling 2 Mini, a 26-billion-parameter model, needs about 16 GiB for it instead of about 96 GiB in fp8 format. The reason: out of 48 passes through the layers, only 8 keep the full cache, while the rest look at just the last 1,024 tokens. For running on your own hardware, that is a sixfold memory saving.
AI-processed from X: @cheese_cakee_9; edited by Hamidun News
Imagine asking a model to analyze an entire code repository or a year's worth of correspondence. That is roughly a million tokens (a token is about a piece of a word). Rather than re-reading everything at each step, the model keeps auxiliary notes for every token.
This memory is called the KV cache, and it usually becomes the ceiling for long context: the weights fit, but the cache for a million tokens balloons to tens of gigabytes. Thinking Machines released Inkling 2 Mini, a 26-billion-parameter model, and according to an architecture breakdown on X, its cache at the full million tokens takes about 16 GiB (gibibytes, nearly the same as gigabytes) instead of roughly 96 GiB if every layer stored everything. The storage format is fp8, meaning 8 bits per number.
Below is where the sixfold difference comes from.
Why Mini exists
The company says some users found the previous Inkling 2 Small too large: it has 594 billion parameters. The answer is Inkling 2 Mini, a looped transformer. Some of its layers are run several times instead of storing separate copies of the weights. It has only 26 billion parameters in total. The company claims the model is small enough to run on a laptop and suits agentic tasks: its score on the Artificial Analysis index (AA Index) is 20.1. The source does not say how this compares with competitors, so the figure should be treated as a reference point, not proof of quality.
Where 16 GiB instead of 96 comes from
The arithmetic is as follows. The network has 24 layers. Twenty of them use a sliding window of 1,024 tokens: each looks only at the last 1,024 tokens, so its cache is tiny and does not grow with the context. The remaining four layers, numbered 5, 11, 17 and 23 (counting from zero), use full attention and remember the entire million tokens.
Now the loop. Layers 11 and 17 sit inside a repeated block and are executed three times each, while layers 5 and 23 run once. That gives 1 + 1 + 3 + 3 = 8 passes with a full cache. There are 48 layer passes in total, so the full cache is needed for only one in six.
For those eight passes in fp8, the result is about 16 KiB per token (roughly 2 KiB per pass). Multiply by a million tokens and you get on the order of 16 GiB. If all 48 passes stored the full cache, the figure would be about 96 GiB. That is the difference between "needs a server GPU" and "may fit in the memory of a powerful workstation."
Note that the loop saves parameters but does not save memory for full attention. In the breakdown, each pass through a layer is counted separately, so layers 11 and 17 are counted three times. The cache reduction comes from the combination: the vast majority of layers look through a narrow window.
What to do with this in practice
First, when planning a local deployment, count not only the weights but also the cache. At fp8, 26 billion parameters come to roughly 26 GB of weights (this is our rough estimate; the company did not publish it), and the cache at the full million tokens adds about 16 GiB. So whether it "runs on a laptop" depends on the memory of the specific machine and the length of the actual context: on short requests the cache will be considerably smaller.
Second, the scheme of "rare full attention plus many local layers" looks like a workable engineering technique for long contexts. The trade-off is the risk that the model captures long-range dependencies in text less well than a network with full attention in every layer. The source contains no independent quality tests at a million tokens: there is only the company's claim and a single AA Index score.
Third, for product teams this is a signal: agentic scenarios with long action histories, large repositories and logs are gradually becoming feasible on local devices. But before building a product on Mini, it is worth waiting for independent tests on real long tasks and comparing it with compact models from other labs.
- Nov 22, 2026EvoNova Labs Releases a Chaos Benchmark of 10 Failure Modes for Always-On AI Agents
- Nov 22, 2026Agent Ignored AGENTS.md: In a Third of Pushes It Didn't Ask, and Force-Pushed 31 Times
- Nov 22, 2026Wells Fargo and four other US banks stop taking responsibility for customers' AI agent mistakes
- Nov 22, 2026Yandex Grant Funds an LLM Engine: 35B Model on RTX 2080 Ti Runs Three Times Faster Than llama.cpp
Key Questions Answered
Why Mini exists?
The company says some users found the previous Inkling 2 Small too large: it has 594 billion parameters. The answer is Inkling 2 Mini, a looped transformer. Some of its layers are run several times instead of storing separate copies of the weights. It has only 26 billion parameters in total. The company claims the model is small enough to run on a laptop and suits agentic tasks: its score on the Artificial Analysis index (AA Index) is 20.1.
Where 16 GiB instead of 96 comes from?
The arithmetic is as follows. The network has 24 layers. Twenty of them use a sliding window of 1,024 tokens: each looks only at the last 1,024 tokens, so its cache is tiny and does not grow with the context. The remaining four layers, numbered 5, 11, 17 and 23 (counting from zero), use full attention and remember the entire million tokens. Now the loop. Layers 11 and 17 sit inside a repeated block and are executed three times each, while layers 5 and 23 run once.
What to do with this in practice?
First, when planning a local deployment, count not only the weights but also the cache. At fp8, 26 billion parameters come to roughly 26 GB of weights (this is our rough estimate; the company did not publish it), and the cache at the full million tokens adds about 16 GiB. So whether it "runs on a laptop" depends on the memory of the specific machine and the length of the actual context: on short requests the cache will be considerably smaller.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.