Starts Nov 23Live Cohort 3 of Vibe Coding & AI Agents (Nov 23, 18:30 MSK): 3 weeks, 9 live hands-on sessionsCohort 3 (Nov 23) →
ReleaseInsight · On RadarX: @cheese_cakee_9→ original

Inkling 2 Mini packs a million-token context into 16 GiB instead of 96

When a neural network reads a long text, it stores notes for every token (a piece of a word) in the KV cache, and at a million tokens that cache swells to tens of gigabytes. Inkling 2 Mini, a 26-billion-parameter model, needs about 16 GiB for it instead of about 96 GiB in fp8 format. The reason: out of 48 passes through the layers, only 8 keep the full cache, while the rest look at just the last 1,024 tokens. For running on your own hardware, that is a sixfold memory saving.

AI-processed from X: @cheese_cakee_9; edited by Hamidun News
Inkling 2 Mini packs a million-token context into 16 GiB instead of 96
Source: X: @cheese_cakee_9. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceX: @cheese_cakee_9 ↗
Editor-in-Chief
Key metric #1Full attention in only 4 of 24 layers; layers 11 and 17 run in a loop three times each
Key metric #2The other 20 layers look only at a 1,024-token window
Key metric #3A score of 20.1 on the AA Index; Inkling 2 Small has 594 billion parameters

Imagine asking a model to analyze an entire code repository or a year's worth of correspondence. That is roughly a million tokens (a token is about a piece of a word). Rather than re-reading everything at each step, the model keeps auxiliary notes for every token.

This memory is called the KV cache, and it usually becomes the ceiling for long context: the weights fit, but the cache for a million tokens balloons to tens of gigabytes. Thinking Machines released Inkling 2 Mini, a 26-billion-parameter model, and according to an architecture breakdown on X, its cache at the full million tokens takes about 16 GiB (gibibytes, nearly the same as gigabytes) instead of roughly 96 GiB if every layer stored everything. The storage format is fp8, meaning 8 bits per number.

Below is where the sixfold difference comes from.

Why Mini exists

The company says some users found the previous Inkling 2 Small too large: it has 594 billion parameters. The answer is Inkling 2 Mini, a looped transformer. Some of its layers are run several times instead of storing separate copies of the weights. It has only 26 billion parameters in total. The company claims the model is small enough to run on a laptop and suits agentic tasks: its score on the Artificial Analysis index (AA Index) is 20.1. The source does not say how this compares with competitors, so the figure should be treated as a reference point, not proof of quality.

Where 16 GiB instead of 96 comes from

The arithmetic is as follows. The network has 24 layers. Twenty of them use a sliding window of 1,024 tokens: each looks only at the last 1,024 tokens, so its cache is tiny and does not grow with the context. The remaining four layers, numbered 5, 11, 17 and 23 (counting from zero), use full attention and remember the entire million tokens.

Now the loop. Layers 11 and 17 sit inside a repeated block and are executed three times each, while layers 5 and 23 run once. That gives 1 + 1 + 3 + 3 = 8 passes with a full cache. There are 48 layer passes in total, so the full cache is needed for only one in six.

For those eight passes in fp8, the result is about 16 KiB per token (roughly 2 KiB per pass). Multiply by a million tokens and you get on the order of 16 GiB. If all 48 passes stored the full cache, the figure would be about 96 GiB. That is the difference between "needs a server GPU" and "may fit in the memory of a powerful workstation."

Note that the loop saves parameters but does not save memory for full attention. In the breakdown, each pass through a layer is counted separately, so layers 11 and 17 are counted three times. The cache reduction comes from the combination: the vast majority of layers look through a narrow window.

What to do with this in practice

First, when planning a local deployment, count not only the weights but also the cache. At fp8, 26 billion parameters come to roughly 26 GB of weights (this is our rough estimate; the company did not publish it), and the cache at the full million tokens adds about 16 GiB. So whether it "runs on a laptop" depends on the memory of the specific machine and the length of the actual context: on short requests the cache will be considerably smaller.

Second, the scheme of "rare full attention plus many local layers" looks like a workable engineering technique for long contexts. The trade-off is the risk that the model captures long-range dependencies in text less well than a network with full attention in every layer. The source contains no independent quality tests at a million tokens: there is only the company's claim and a single AA Index score.

Third, for product teams this is a signal: agentic scenarios with long action histories, large repositories and logs are gradually becoming feasible on local devices. But before building a product on Mini, it is worth waiting for independent tests on real long tasks and comparing it with compact models from other labs.

Story

Key Questions Answered

Why Mini exists?

The company says some users found the previous Inkling 2 Small too large: it has 594 billion parameters. The answer is Inkling 2 Mini, a looped transformer. Some of its layers are run several times instead of storing separate copies of the weights. It has only 26 billion parameters in total. The company claims the model is small enough to run on a laptop and suits agentic tasks: its score on the Artificial Analysis index (AA Index) is 20.1.

Where 16 GiB instead of 96 comes from?

The arithmetic is as follows. The network has 24 layers. Twenty of them use a sliding window of 1,024 tokens: each looks only at the last 1,024 tokens, so its cache is tiny and does not grow with the context. The remaining four layers, numbered 5, 11, 17 and 23 (counting from zero), use full attention and remember the entire million tokens. Now the loop. Layers 11 and 17 sit inside a repeated block and are executed three times each, while layers 5 and 23 run once.

What to do with this in practice?

First, when planning a local deployment, count not only the weights but also the cache. At fp8, 26 billion parameters come to roughly 26 GB of weights (this is our rough estimate; the company did not publish it), and the cache at the full million tokens adds about 16 GiB. So whether it "runs on a laptop" depends on the memory of the specific machine and the length of the actual context: on short requests the cache will be considerably smaller.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…