LexFrame reads LLM hidden states: 0.97 coherence against a lens ceiling of 0.30

Researchers have proposed a way to read a language model's hidden states the way the model itself reads them. The method is called Lexical Reference Frames. At mid-depth on SmolLM3-3B and OLMo3-7B, any lens with fixed per-layer directions is capped at 0.30 and 0.35. The first-order version of the method, LexFrame[1], reaches 0.97 coherence on both models. The state it reads differs from the model's actual output by no more than 0.16 nats.

Why fixed lenses hit a ceiling

The model's vocabulary defines its native semantics. The embedding and the unembedding fix the meaning of a state at the two ends of the residual stream. Between them, existing approaches fall into two camps. One takes a fixed lens and applies the same directions to states at a given layer. The other treats states as a point cloud and looks for structure in it. The authors argue that neither approach reads a state the way the model does.

Their argument is that every hidden state has its own frame of reference. The layers that will read that state next define the frame, and it adapts to context. A fixed lens necessarily loses this context dependence, because its directions are set in advance. That yields a formal bound: a lens with fixed per-layer directions cannot exceed the square root of the native frame's coherence. At mid-depth, that is 0.30 for SmolLM3-3B and 0.35 for OLMo3-7B.

How LexFrame works

A lexical reference frame approximates the native frame of reference. The state is passed through the model's own remaining layers. A set number of attention rounds is run between positions, and a typical attention output is substituted for all other contributions. There is no fitting to the model's output, and no coefficients are trained on its predictions. In the LexFrame[1] notation, the index, judging by the description, corresponds to the number of attention rounds, so this is a first-order frame.

The result is read in a lexical basis, that is, in terms of vocabulary tokens. Every intermediate state therefore gets an interpretation in the same units the model uses to produce its answer.

What the measurements showed

The authors compared LexFrame[1] with lenses and with attribution patching on two models. The reported results are:

• LexFrame[1] coherence at mid-depth: 0.97 on SmolLM3-3B and 0.97 on OLMo3-7B;
• ceiling for any lens with fixed directions: 0.30 and 0.35;
• divergence from the model's output: within 0.16 nats;
• attribution of predictions via frame coefficients is more accurate than attribution patching;
• early exit based on confident readings lets the model skip 0.45 of the layers on SmolLM3-3B and 0.42 on OLMo3-7B, whereas lenses skip almost none.

The authors also note separately that the lexical basis makes it possible to steer a prediction without affecting the other tokens. That sets it apart from a crude activation shift, which usually smears the whole distribution.

Where this is useful in practice

For interpretability, the key point is that one tool handles three tasks: state reading, attribution and steering. Previously these required different methods with different assumptions.

For inference, early exit is the more interesting part. If a confident reading at an intermediate layer matches the model's answer, the remaining layers can be skipped. By the reported figures, that is almost half the depth on the two models tested. For production, savings of that size would matter, but the work has caveats. The authors tested only 3B and 7B models, and the abstract measures early-exit quality by closeness to the model's output, not by end-to-end task metrics. Before counting on a speedup in a real service, you need to look at the code and at how the method behaves on larger models and longer contexts.

Safety and debugging researchers should take a look at the targeted steering. If the lexical basis really changes one prediction and leaves neighboring tokens alone, it could be used for local behavior edits and for testing causal hypotheses about how the model arrives at its answer.