LLM agents and SOC: developers build a RAG-based cyber threat database

Developers have described on Habr how they built their own cyber threat database for LLM agents and SOC systems. The service combines PDF documents, CVE records and cybersecurity articles in a single database and serves relevant fragments to an AI agent via an HTTP API.

How the database pipeline works

At first, the task looked linear and fit into a single chain of steps. The sources (PDF files, CVE vulnerability descriptions and information security articles) are first ingested into the system, and then the text is extracted from them.

• Data sources: PDF documents, CVE records and cybersecurity articles
• The text is split into fragments (chunking) for subsequent indexing
• Vector representations (embeddings) are built from the fragments for semantic search
• The finished database serves results to the LLM agent via an HTTP API
• The authors built a working prototype quickly; the main work began after it launched

After extracting the text, the system splits documents into parts, builds vector representations from them and turns the result into a search index. An LLM agent or SOC analyst then pulls context on a specific threat from that index.

Why the prototype is only the beginning

According to the authors, a working prototype came together quickly, and the real work started afterward. This is a common pattern for RAG (retrieval-augmented generation) knowledge bases in specialized domains like cybersecurity. You can assemble the basic chain of ingestion, text extraction, chunking, vectorization and search in a short time. Turning it into a source that an LLM agent and a SOC analyst actually trust is a problem on an entirely different scale.

This domain has an added difficulty: heterogeneous formats. PDF threat reports, structured CVE records and ordinary articles are built differently. Each needs its own approach to text extraction and chunking before everything can be merged into a single search index.

Why LLM agents and SOC need this

The idea is to give the LLM agent current, verifiable fragments from PDF reports, CVE records and security articles via an HTTP API, rather than general knowledge from its training data. For a SOC (Security Operations Center) analyst, this means the model's answer about a specific threat rests on a real source, not only on what the model "remembers" from pretraining. The industry calls this approach retrieval-augmented generation, or RAG.

That is why the authors frame the effort not as a one-off build but as an ongoing process. The cyber threat database has to be continuously updated with new CVEs and articles, so the ingestion, chunking and search pipeline needs ongoing maintenance and improvement even after the first version starts answering queries.

The five-step scheme (ingestion, text extraction, chunking, embedding generation, search) is typical of modern RAG systems and follows a clear linear logic that is easy to sketch on a whiteboard. This simple version of the pipeline is exactly what the authors built first, before moving on to the more labor-intensive refinement stage.

What it means

The case shows a typical path for building RAG systems in narrow professional domains. You can put together a working pipeline for ingestion, chunking and vector search quickly, but turning it into a tool that LLM agents and SOC analysts trust is a task that only truly begins after the first prototype.