Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseInsight · On RadarHabr AI→ original

RuBERT Taught to Use a Knowledge Graph: Precompute the Vectors, Leave Attention Alone

RuBERT sees which words appear near each other, but it does not know that "gradient descent" is a type of optimization, so it stumbles on rare scientific terms. The author of a Habr write-up proposes precomputing concept vectors from a knowledge graph (a database of links between terms) and concatenating them with the RuBERT vector before a small classifier. This avoids the heavy K-BERT and its 512-token limit. There are no quality measurements yet, so you will have to test it on your own data.

AI-processed from Habr AI; edited by Hamidun News
RuBERT Taught to Use a Knowledge Graph: Precompute the Vectors, Leave Attention Alone
Source: Habr AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
Key metric #1K-BERT inflates the input with triplets and runs into the 512-token limit
Key metric #2A key-based vector lookup costs O(1) per concept, and no graph neural networks are needed
Key metric #3A wrongly linked term is just one vector in an average, not a failure of the whole text

Imagine you are building a rubricator for a scientific library or an internal article database. You take RuBERT, the Russian-language version of the BERT language model from DeepPavlov, and fine-tune it on labeled texts. As long as the terms are simple, quality is decent.

But then interdisciplinary papers arrive with rare concepts, and the model gets confused: it knows which words often appear together, but it does not know that "gradient descent" is a kind of optimization. A write-up on Habr describes how to close this gap without a heavy rebuild of the model: precompute concept vectors from a knowledge graph (a database where the links between terms are recorded explicitly) and mix them into RuBERT's output. One caveat up front: this is a design for a method, not a report on experiments.

The material contains no quality figures, which is why the headline has none either.

Why Not K-BERT or ERNIE

The most obvious route is to build the knowledge graph directly into the transformer. K-BERT finds entities in the text, pulls their "subject — relation — object" triplets and attaches them to the sentence, turning it into a tree. To keep attention from spreading across irrelevant branches, it adds special position indices and a visibility matrix.

This approach has three weak spots for long scientific papers. The sequence grows, and the cost of the attention mechanism grows quadratically. BERT's limit of 512 tokens (roughly a few paragraphs of text) runs out faster, and the article's context has to be truncated. And if a term is linked to the wrong graph node, the error travels through attention and corrupts the representation of the whole text.

ERNIE works differently: during pretraining it masks entire named entities and pulls in their descriptions from a knowledge base. But the structure of the graph stays out of view: the model does not see the hierarchy or the types of links. On top of that, narrow terms often have no encyclopedic descriptions at all.

How the Lightweight Scheme Works

The idea is to leave the internals of the language model untouched. The knowledge graph is converted into vectors in advance, offline, using one of several methods. TransE treats a relation as a shift in vector space. RotatE treats it as a rotation in complex space and captures symmetric and inverse relations well. node2vec uses random walks over the graph and does not require labeled relation types.

Then come two streams. The text stream: RuBERT outputs the vector of the special [CLS] token, 768 numbers long, which captures the meaning of the whole document. The graph stream: an NER module (named entity recognition) finds scientific concepts in the article, each is linked to a graph node, and their vectors are fetched from a frozen table by key and averaged. If no concepts are found, a zero vector is substituted, and the classifier relies on the text alone.

The two vectors are concatenated and fed into a small MLP classifier: a hidden layer of 256 neurons, a ReLU function, dropout of 0.3 and a softmax output. In the PyTorch example, the entire graph part is nn.Embedding.from_pretrained(..., freeze=True) and a few lines of masked averaging. Training uses cross-entropy with the AdamW optimizer. This fusion at the level of ready-made features, rather than raw data or final outputs, lets the classifier learn the interaction between context and graph structure.

What This Gives in Practice and Where the Catch Is

The economics are simple. A key-based vector lookup costs O(1) per concept, meaning it does not depend on the size of the graph. Heavy graph neural networks are not needed at runtime, and the attention mechanism does not change. No tokens are spent on knowledge, so neither input length nor speed suffers noticeably. Robustness is also better: a wrongly linked term is one vector out of many in an average, not a poisoned context.

There are plenty of catches. First, everything depends on the quality of NER and of linking terms to nodes: without it, the graph adds nothing. Second, you need a ready-made graph that covers your domain, otherwise most articles will get a zero vector. Third, BERT's 512-token limit has not gone anywhere: long articles will still have to be cut into fragments, or you will have to use the title with the abstract. Fourth, averaging erases the order and importance of terms, and a key concept can get drowned out among minor ones.

The author plans to compare the method on Macro F1 (a metric sensitive to rare classes in imbalanced data), as well as on Micro F1, accuracy and milliseconds per document. The baselines are TF-IDF with logistic regression or SVM, plain RuBERT, K-BERT and ERNIE. The success criterion is a Macro F1 gain over plain RuBERT at a higher speed than the heavy hybrids.

The practical takeaway: if you have an ontology or a domain graph and a working NER, you can assemble the scheme in a couple of days. First compare it with plain RuBERT on your own data, and only then decide whether you need the graph. For now it is a hypothesis, and you can only believe it after your own measurements.

Story

Key Questions Answered

Why Not K-BERT or ERNIE?

The most obvious route is to build the knowledge graph directly into the transformer. K-BERT finds entities in the text, pulls their "subject — relation — object" triplets and attaches them to the sentence, turning it into a tree. To keep attention from spreading across irrelevant branches, it adds special position indices and a visibility matrix. This approach has three weak spots for long scientific papers. The sequence grows, and the cost of the attention mechanism grows quadratically.

How the Lightweight Scheme Works?

The idea is to leave the internals of the language model untouched. The knowledge graph is converted into vectors in advance, offline, using one of several methods. TransE treats a relation as a shift in vector space. RotatE treats it as a rotation in complex space and captures symmetric and inverse relations well. node2vec uses random walks over the graph and does not require labeled relation types. Then come two streams.

What This Gives in Practice and Where the Catch Is?

The economics are simple. A key-based vector lookup costs O(1) per concept, meaning it does not depend on the size of the graph. Heavy graph neural networks are not needed at runtime, and the attention mechanism does not change. No tokens are spent on knowledge, so neither input length nor speed suffers noticeably. Robustness is also better: a wrongly linked term is one vector out of many in an average, not a poisoned context. There are plenty of catches.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…