G-Retriever's Graph Token Turns Out to Be a Dud: Cosine Similarity of 0.985–0.998

The graph token that G-Retriever and its successors train a GNN to produce carries almost no information about the graph. That is the conclusion of a group led by Andrea Giuseppe Di Francesco at Sapienza University of Rome, together with Rishabh Jain, Franco Maria Nardini, Nicola Tonellotto, Pietro Liò and Fabrizio Silvestri. Cosine similarity between tokens from different examples is 0.985–0.998, which means that to the LLM it is practically the same vector. The authors call the phenomenon token collapse and show that accuracy on graph question-answering tasks does not rest on the token at all.

How they tested it: token swaps and a constant in place of the GNN

G-Retriever belongs to the family of graph soft prompting methods. A GNN encodes a knowledge subgraph into one or more soft tokens, which are added to the prompt. In parallel, the same graph is written out in words and passed in as ordinary text. The assumption was that the token conveys relational structure and the text complements it.

The authors took this setup apart with interventions. They replaced the token with:

• the dataset-mean token;
• a token from a different example;
• a token computed from a different or corrupted graph.

None of the replacements noticeably affected answer accuracy. The authors then trained a constant soft prompt that never touches the graph. On all three benchmarks it matched or outperformed G-Retriever. In other words, the token works as an ordinary learnable soft prompt, and the graph reaches the model through its text verbalization.

The scale of the problem: not one method and not one configuration

The collapse does not depend on a lucky choice of components. It was observed on three benchmarks, with three graph encoders, on two LLM families, and with both frozen models and models adapted via LoRA. Follow-up methods, both single-token ones and multi-token ones with pooling, inherited the same flaw.

The authors are cautious about linking the collapse to architecture and training: it is associated with the Sigmoid projector and with end-to-end training on answers. When the only signal is the correct answer and the text channel already supplies everything needed, it is easier for the optimizer to learn a constant than to extract structure from the graph.

A diverse token is not necessarily a useful one

The authors proposed a two-stage pipeline that produces non-collapsing tokens and, in most cases, retains competitive accuracy. But diverse vectors do not prove that the model uses the graph. Interventions and decoding probes showed that, depending on the configuration, the token may encode:

• the question itself;
• the prior distribution of answers;
• graph information that the LLM ignores;
• graph information that actually changes the answer.

The last option is the one the method was designed for, and it has to be verified separately.

The main practical outcome of the work is a protocol that directly tests the correspondence between token and graph, rather than inferring it from final accuracy. The code to reproduce the experiments is published in the Token_Collapse_KGQA repository on GitHub.

Teams building GraphRAG and KGQA systems on soft graph tokens should add three checks to their evaluation. First, replace the token with the dataset mean and compare metrics. Second, train a constant prompt as a baseline. Third, measure the cross-example cosine similarity of the tokens. If the token can be swapped out without loss, the graph's contribution to quality flows through the text, and the GNN in such a setup merely burns compute. Those who still want a genuine graph signal should try two-stage training and decoding probes to tell a token that carries graph information from one that simply repeats the question.