Strata Runs Qwen3.8-Flash-Next on a 12 GB RTX 5070: 94 Tokens per Second at Q2_0

The Strata project runs the MoE model Qwen3.8-Flash-Next on an RTX 5070 with 12 GB of memory and reaches 94 tokens per second. That result comes from the Q2_0 quant (weights compressed to roughly two bits) on a system with a Ryzen 5 7600 CPU and 64 GB of RAM. The secret is a memory hierarchy modeled on a processor with caches: VRAM, RAM and SSD play the roles of L1, L2 and L3. Mixture of Experts activates only about ten of the 24,576 experts per token, yet all the weights still have to be stored. That is the problem Strata solves.

Three memory tiers instead of VRAM alone

MoE sparsity cuts the compute per token, but not the bytes that must be kept on hand for the model to run. A dense model of the same size reads all of its weights from memory at every step, while an MoE reads only a small fraction. Strata lays out the data as follows:

• L1, video memory. Attention layers, DeltaNet, routers, the most in-demand experts, the MTP layer, the KV-cache and a cache of popular experts.
• L2, system RAM. All 24,576 experts, which the CPU processes in parallel with the GPU.
• L3, disk. An n-gram table several tens of gigabytes in size. Only a handful of rows are read from it per token, so SSD speeds are enough.

Hybrid schemes have been tried before, but they ran into PCIe bandwidth limits. Strata sidesteps the limit: if the needed expert is not in VRAM, it is not copied over the bus but computed on the CPU directly in system RAM. As a result, VRAM capacity still matters more than GPU power: each gigabyte of VRAM adds roughly 700 experts to the fast cache. The usual optimizations work too: MTP (predicting several tokens per step) and prefetching the next layer's experts while the current layer's attention is being computed.

Models, quants and speed

The project currently supports Qwen3.8-Flash-Next, Qwen3.6-35B-A3B and DeepSeekV4.1-Flash-1017. The best quants come from the Distributed Algorithms and Systems lab at the Institute of Science and Technology Austria, which develops the GSQ/RCO quantization methods and publishes its results on HuggingFace. The choice of quant now depends on the sum of RAM and VRAM. For Qwen3.8-Flash-Next (the source gives both 125 billion and 180 billion parameters), the thresholds are: Q2_0 — about 38 GB, IQ2_XS — about 40, IQ3_XXS — about 47, IQ3_S — about 55 GB. For DeepSeekV4.1-Flash-1017 (763 billion parameters): IQ1_M — about 46 GB, IQ2_XXS — 54, IQ2_XS — 64 GB.

Quality retained relative to the BF16 baseline: IQ3_S — 90.0%, IQ3_XXS — 87.3%, IQ2_XS — 85.0%, Q2_0 — 84.1%, and UD-Q4_K_XL — 94.1%.

The speeds, which the author collected from around the internet, depend on the configuration. On an RTX 5070 12 GB, the Q2_0, IQ2_XS and IQ3_S quants correspond to 94, 79 and 53 tokens per second. An RTX 4070 with 32 GB of RAM and IQ2_XS delivers 48, Strix Halo with UD-IQ4_XS delivers 40, and the integrated Radeon 780M with Q2_0 and MTP delivers 14. The author's own measurement: IQ3_S on an RTX A6000 48 GB, 128 GB of RAM and a Threadripper 3970X with a 256K context reaches 150 tokens per second.

DeepSeekV4.1-Flash-1017 is noticeably slower: from 5 tokens per second on an RTX 5070 with IQ2_XXS to 25 on an RTX 4090 with IQ2_XS. The author's measurement on an A6000 with IQ2_XS is 30 tokens per second.

Project Maya and the price of the breakthrough

Aggressive optimization of the engine for quants closed the door to models from other families, so the Project Maya fork appeared. It runs GLM-5.3-Flash (321 billion), DeepSeek-V4.1-Flash-1017 (748 billion) and MiniMax-M3.1-Flash (428 billion). GLM-5.3-Flash is available in an abliterated, uncensored variant, which is handy for reverse engineering, and a context of up to 1M allows long sessions without compression.

The authors released four quants; accuracy is given relative to FP8:

• Maya-S, 96.5 GB, for cards up to 16 GB, 97.9%;
• Maya-S24, 94.7 GB, for cards with 16–24 GB, 97.7%;
• Maya-M, 116 GB, for cards with 24–48 GB, 97.9%;
• Maya-L, 156.3 GB, for cards with 48 GB or more, 99.2%.

The minimum to run is 32 GB of RAM. On two H100 80 GB cards, Maya-L with a 1M context delivers 122–124 tokens per second. S24 on an RTX 3090 with 128 GB of RAM delivers 26.0, on an RTX 5090 23.0, and on an RTX 4090 with a 32K context 16.6. The author's measurement for Maya-M on an A6000 with a 1M context is 28 tokens per second.

The breakthrough has a price: quantization down to 2–3 bits, 64–128 GB of fast RAM and fast NVMe drives. Most of the figures were gathered from public sources rather than measured with a single method, so compare the table rows with caution. The question "will the model fit" has given way to "how many bits of quantization are you willing to give up for speed," and the answer depends on the task.