Five Ways to Speed Up MoE on a GTX 1070: Auto-Prefix Cut Latency from 13.3 to 6.1 s

If you are trying to run a large MoE model on a home GPU, the bottleneck is memory, not compute. Expert weights do not fit in VRAM, so some spill over to RAM and SSD, and every access to a "cold" expert costs precious milliseconds. The author of the Strata engine, who runs Qwen3.8-Flash-Next on a GTX 1070 with 8 GB, has put together five techniques that help with this. Four are already built into the engine. He wrote the fifth himself and published measurements.

Four techniques already in the engine

First is speculative decoding (--spec N --mtp <path>). A small draft layer predicts several tokens ahead, and the large model verifies them in a single pass. If the draft guesses right, you get several tokens for the price of one. The gain is usually 1.3–1.6x. The draft guesses code and repetitive patterns well, and free-form prose less well. The cost is a bit of VRAM for the draft layer and buffers.

Second is --kv-resident N. Only a sliding attention window of N tokens stays in VRAM, and the entire KV-cache moves to pinned RAM. This does not speed up attention itself. The gain comes from elsewhere: the freed-up memory goes to the expert cache. For an MoE, an extra gigabyte for resident experts is usually worth more than the savings on KV. It is easy to check: compare the expert cache hit rate in the log before and after.

Third is batching (--batch, --parallel). This is continuous batching: several conversations are decoded in shared windows, and the shared weights are read once for everyone. On a single card, per-user throughput even drops slightly. In return, nobody waits in a queue while someone else is being served. If you have several GPUs with layer-split or parallel agents, see docs/BATCHING.md.

Fourth is adaptive expert residency (--adapt-every, --adapt-swaps). The cache moves weights between VRAM, RAM, and SSD by access frequency on its own, during generation. By the author's measurements, the difference between a static and a "live" cache was 1.5x to 2x, depending on how uniform the prompt stream was.

Fifth technique: automatic common prefix

Strata has a strata_prefix mechanism: the client explicitly specifies where the shared part of the prompt ends, and the engine stores a checkpoint at that boundary. Subsequent requests with the same beginning do not re-read it. The problem is that almost no client, neither chatbot nor agent framework, does this.

The author added a server-side layer (STRATA_AUTO_PREFIX=1, off by default). It keeps a radix-trie over the token IDs of recent prompts and maintains a budget for the number of simultaneous "pins". Each pin takes VRAM from the expert cache; there are no free pins. If a request arrives without an explicit prefix, the server finds the match itself and inserts the boundary. The engine does not notice the difference.

Measurement: Strata 0.2.12, one GTX 1070 8 GB, Qwen3.8-Flash-Next UD-IQ4_XS, two clients with one long document.

• Median re-read tokens: 886 → 468.
• Median tokens served from cache: 0 → 413.
• Median time to first token: 13.3 s → 6.1 s.

These are two runs of the same benchmark on the same machine, with only one environment variable changed.

Limitations and what to do

First, the caveats. All the numbers come from a single configuration and a single scenario with two clients, and the gain from auto-prefix depends on how much of a beginning your requests share. The patch lives in a fork branch and has not been accepted upstream. The author opened an issue rather than a PR because CONTRIBUTING.md requires agreeing on scope first. Pins compete with the expert cache for VRAM, so you will have to tune the budget.

The practical order is as follows. If you run an MoE on a modest card, start with --kv-resident and --spec/--mtp: this is the fastest win without a single line of code. Then check that expert adaptation is enabled. If you have RAG or agents with a long shared system prompt, auto-prefix is worth testing on your own traffic. The main lesson is broader than one engine: with limited VRAM, almost any optimization comes down to the question of what to spend each gigabyte on. You have to verify that with measurements, not intuition.