Token position recovered at 99% from a NoPE transformer, but only 29–42% from RWKV-7
The author trained 12 small models (RWKV-7, transformers with RoPE, and transformers with no positional encoding, or NoPE) and ran a linear probe on every layer. The probe recovered a token's position in the text with 99% R² from NoPE, 38–51% from RoPE, and 29–42% from RWKV-7. At equal loss, syntax and meaning are readable about equally well in all of them. So compare models at the same loss: otherwise a "win" may come from lower error alone, not from the architecture.
AI-processed from Habr AI; edited by Hamidun News
Imagine you are comparing two versions of a RAG config and the new one answers more accurately. Don't celebrate yet: it may simply have been tuned for longer and so scores better on every metric, with the idea itself playing no part. An independent researcher ran into exactly this problem while comparing the RWKV-7 architecture (a recurrent model without an attention mechanism) with ordinary transformers.
He trained 12 small models that differ only in architecture and looked inside every layer of every checkpoint. The main result: at this scale, the clearest difference is how the models store the position of a token, that is, the number of a word in the text. For grammar and meaning, the models look similar at the same training error (loss).
The headline figures of 99% and 29–42% are the R² metric (how accurately a simple model guesses a token's number from a layer's hidden state; 100% is perfect).
What was compared and how
The study used three ways of giving a model word order. RWKV-7 keeps a recurrent state, and order is built into how that state updates. A transformer with RoPE gets position by rotating queries and keys inside attention. A NoPE transformer has no positional encoding at all: its only source of order is the causal mask, which forbids looking at words ahead.
There are two sizes (56.6M and 115M parameters), two languages (Russian and English), and 1–2 billion tokens per model. The corpus is roughly 76% Wikipedia. Each run saved 21 checkpoints, and 9 of the 12 runs have a second initialization seed.
The tool is linear probes. A probe is a simple model that tries to guess a property from a token's frozen hidden state: position number, role in the sentence, case, verb tense, sentiment. If the probe succeeds, the information sits in the layer "in plain sight." The author's caveat: a probe shows what can be read out, not what the model actually uses.
Position is stored differently
The best NoPE layer reached an R² of 99%, RoPE 38–51%, and RWKV-7 29–42%. The ordering is the same across both languages, both sizes, and both seeds. NoPE reaches an R² of about 98% by the third layer and holds at 85% or above until layer 11, then drops to 22–44% at the last layer.
One curious detail: even a randomly initialized transformer reads position with an R² of 44–56%, because the causal mask alone provides a signal. In RoPE this figure falls during training: in the Russian 56.6M model, from 51% to 38%. The author suggests that since position is fed directly into attention, there is no longer any need to store it in the residual stream, but he did not test this.
A word's depth in the dependency tree shows the same ordering, but the gap is much smaller, 2–6 percentage points. Depth is tied to a word's number in the sentence, so the author treats it as the same effect rather than a separate finding.
Why comparing at equal loss matters
RWKV-7's final validation loss is 0.10–0.13 lower than RoPE's, so comparing the final checkpoints is unfair: a model with lower error looks "smarter" on almost any probe. For example, at the final checkpoint RWKV is better at telling Russian homonyms apart (87.6% versus 81.9%), but at equal loss this lead shrinks or vanishes. So the author interpolated probe results by loss across the 21 checkpoints and compared the architectures at four loss levels. A difference counted as robust only if it appeared in both seeds, had the same sign at all levels, and exceeded the spread between seeds.
The remaining differences are small, 0.3–3 percentage points. NoPE is slightly weaker on a word's role in the sentence, and on verb tense and aspect (by 0.4–2.3 pp), while RoPE and RWKV show practically no differences. In the English models, RWKV is slightly ahead of NoPE on homonyms and sentiment, but the test sets are tiny (16 words per sense), and the author considers the signal weak.
There is also a lead worth following: the syntactic probes peak 1–4 layers lower in RWKV than in the transformers. But the curves are flat near the maximum, and the spread of the peak layer number between seeds was not estimated.
What to do in practice
The lesson is above all methodological. When comparing architectures, checkpoints, or configs, hold loss fixed rather than step count or model size, and don't trust the claim "architecture X understands Y better" without that control.
The limitations are serious: the models are small, there are only two seeds and they change only the initialization, data order was not varied, the author wrote the semantic tests himself, and he checked for data leakage only for Russian syntax. The result says nothing about models larger than 115M, or about which architecture makes better use of position.
Everything is open: the probe code and results are on GitHub under Apache 2.0, and all 21 checkpoints of each of the 12 seed 0 runs (86.5 GB) are on Hugging Face. The training code and configs have not yet been published, so the probes can be reproduced right away, but the training cannot.
- Nov 8, 2026RWKV-7's system prompt can be “folded” into a state: 12 times faster startup
- Oct 28, 2026rwkv-quant: keeping one matrix in bf16 removes 39% of RWKV-7 2.9B's quantization loss
- Oct 29, 2026T-Tech Introduces LIMe, a Method That Trains Language Models With 15% Less Compute
- Nov 12, 2026Static embeddings processed 32.9 million texts on a laptop in 35 minutes
Key Questions Answered
What was compared and how?
The study used three ways of giving a model word order. RWKV-7 keeps a recurrent state, and order is built into how that state updates. A transformer with RoPE gets position by rotating queries and keys inside attention. A NoPE transformer has no positional encoding at all: its only source of order is the causal mask, which forbids looking at words ahead. There are two sizes (56.6M and 115M parameters), two languages (Russian and English), and 1–2 billion tokens per model. The corpus is roughly 76% Wikipedia.
What are the key details on Position is stored differently?
The best NoPE layer reached an R² of 99%, RoPE 38–51%, and RWKV-7 29–42%. The ordering is the same across both languages, both sizes, and both seeds. NoPE reaches an R² of about 98% by the third layer and holds at 85% or above until layer 11, then drops to 22–44% at the last layer. One curious detail: even a randomly initialized transformer reads position with an R² of 44–56%, because the causal mask alone provides a signal. In RoPE this figure falls during training: in the Russian 56.6M model, from 51% to 38%.
Why comparing at equal loss matters?
RWKV-7's final validation loss is 0.10–0.13 lower than RoPE's, so comparing the final checkpoints is unfair: a model with lower error looks "smarter" on almost any probe. For example, at the final checkpoint RWKV is better at telling Russian homonyms apart (87.6% versus 81.9%), but at equal loss this lead shrinks or vanishes. So the author interpolated probe results by loss across the 21 checkpoints and compared the architectures at four loss levels.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.