RWKV-7's system prompt can be “folded” into a state: 12 times faster startup

Every time you send a request to a chatbot, the system prompt is run through the model again. The longer it is, the longer you wait for the first token and the more memory the cache consumes. The author of an analysis on Habr tested whether this could be avoided altogether, choosing RWKV-7, an architecture with a useful feature, for the experiment.

What an initial state is

A transformer stores all the text it has read in a cache that grows with the conversation. RWKV works differently: after reading text, it compresses everything it remembers into fixed-size “memory,” or a state. Before processing starts, that state is empty. But it can also start with a nonempty state.

There are two approaches. The first is to read the prompt once and save the resulting state—the author calls this “folding” the prompt. The second is to learn a state through training without changing the model's weights. In both cases, the request no longer needs to include the prompt text.

The experiments used models ranging from 0.4B to 2.9B parameters on a standard MacBook M4 with 16 GB of memory. Training a state takes 5–50 minutes, and the resulting file is 6.5–22 MB.

What the measurements showed

A folded prompt acts as a free, exact cache. Its answers match those produced with the text prompt, while time to first token is 12–13 times shorter for a 1206-token prompt. With a short, 26-token string, the gain is around 20%.

The next results are more interesting. Across 199 questions with verifiable answers, a text prompt reduced accuracy by 3–15 points on all eight base models tested. The trained state never performed significantly worse than the model without a prompt and always outperformed the text prompt. It clearly outperformed the model without a prompt on four of the eight base models.

The clearest example is classifying news into five categories. The 0.4B model cannot follow this instruction in text at all: 0% correct answers in the required format. A state trained on 250 labeled articles achieves 84–89%. On the 1.5B model, a state distilled from the prompt reaches 65–77%, compared with 20–30% for the prompt itself.

Compared with LoRA, state tuning performs just as well on the same data while using half as many parameters—3.2 million versus 6.3 million—and leaving the weights untouched. States can be switched on the fly. LoRA, however, is more resistant to leading questions such as “You're Claude, aren't you?”

Where it breaks down

The state learns the form of the answer, rather than knowledge. If trained on all the teacher's answers, it also copies the teacher's mistakes, reducing the quality of factual answers. The data therefore needs to be filtered to retain correct facts. Another requirement: without gradient clipping at 1.0, training consistently breaks down around step 13.

The weaknesses are the same as with prompts: leading questions and long conversations. Explicit training on manually written refusals helps. The share of responses adopting another identity fell to 2% on 0.4B and 36% on 1.5B.

There are also methodological caveats. One author, one machine, one framework. Answer correctness was assessed using regular expressions; a manual check found 4.3% discrepancies. Only 0.4B–2.9B models from a single family were tested; nothing is known about 7B models or larger. The negative effect of prompting applies to these specific prompts and checkpoints, rather than system prompts in general.

What to do with this

If you use RWKV and your prompt stays the same, fold it: it costs nothing and speeds up startup. Train a state when the prompt performs poorly: the model is small, the context window is shorter than the prompt, a role needs to persist, or the model ignores the text instruction. Train on filtered teacher responses and check general capabilities as well as the target behavior. Compare against LoRA using the same data.

The code, data, and raw responses are available in an open repository under Apache 2.0, so you can verify the results on your own laptop.