AnythingLLM Builds a Private RAG Assistant Over Your Documents in One Evening

A familiar situation: your disk holds a hundred PDFs of contracts, lecture notes and internal documentation. You upload one file to a chatbot and it copes. You upload twenty, and the context overflows and part of the data gets lost. Add a subscription, a VPN and a reluctance to hand your documents to someone else's cloud. AnythingLLM was built for exactly this task.

What it is and how it works

AnythingLLM is a full-stack open-source application. It takes your files, splits them into chunks, vectorizes them, stores them in a database and hands the model only the relevant pieces. There is no shortage of formats: PDF, DOCX, TXT, web pages, and audio via Whisper.

The main trap for beginners is that there are two ways to add a document. If you "attach" a file to the chat, its full text goes into the context window. That is fast, but a couple of large files and the window is full. Embed mode works like proper RAG: only the fragments close to the question end up in the prompt.

Documents live in a workspace. These are isolated spaces for projects, each with its own memory, its own files and its own model. One workspace can run on a local Ollama, another can call Claude through the API. Version 1.13.0 added Model Router: it sends simple requests to a cheap model and complex ones to a flagship. You define the rules, for example by keywords or request size. Scheduled Jobs, meaning agents that run on a schedule, also appeared there.

The default vector database is the built-in LanceDB. If needed, you can connect PGVector, Chroma, Milvus, Pinecone, Qdrant and others.

Desktop or Docker

The desktop version installs in one click on Mac, Windows and Linux. Everything runs on your machine, and the built-in model offers three GGUF variants suited to your hardware. It is a good way to figure out in ten minutes whether you need the tool. But this is a single-user mode, and with a large document base the built-in engine starts to struggle.

Docker is for a server: multi-user mode, invitations, workspace access control, chat widgets for a website, and the full API. A backup comes down to copying a single storage directory. Two common pitfalls:

• the container runs as a user with UID/GID 1000, and if the host folder belongs to root, SQLite cannot create the database and the container goes into an endless restart loop. It is fixed with chown -R 1000:1000 on the storage directory;
• the --cap-add SYS_ADMIN flag looks suspicious, but without it the built-in headless browser for web pages will not start.

An external model is connected through the Generic OpenAI provider: you just need to specify the Base URL, the key and the exact model ID. This is how Russian gateways such as Bothub connect, as well as LM Studio, Ollama in compatibility mode, vLLM and OpenRouter. The scheme is the same, only the address changes. Two fields are worth filling in right away: Token context window, set to the model's real window, and Max tokens. The default there is 1024, and long answers will be cut off without warning.

Where it hurts

First, RAG quality. Out of the box it works, but only so-so. The result depends on chunk size, the embedding model and the similarity threshold, and the defaults are a compromise rather than tuning for your task. A typical symptom: the model answers from one paragraph in the middle of a PDF and ignores three others that say the same thing in different words. Your first evening will go to figuring this out.

Second, agent stability. In 2026 there were memory leaks: each RAG request in an agent session added from 100 to 650 MB that was never freed. That is tolerable for home experiments, but not for a server with a dozen users.

Third, security. CVE-2026-48116 allowed arbitrary command execution inside the container via argument injection in ripgrep. The file plugin is enabled by default in the official image, so it was enough to simply send the agent a message. CVE-2026-45403 gave path traversal through symbolic links. Both are fixed in 1.13.0, and the current version at the time of writing is 1.19.0. In addition, the desktop version has no authentication, and the Docker version does not ask for a password until you enable one yourself. An open port without a password is not privacy, it is a lottery.

I do not recommend the cloud hosted version at 50 dollars a month for serious document work: it has no built-in LLM, the embedder runs on CPU with strict limits, and custom agents and MCP are disabled.

Practical takeaway

If you have a corpus of texts and do not want to hand it to public clouds, AnythingLLM gives you a working foundation in one evening. Start with the desktop, move to Docker for a team, enable a password right away, update to the latest version and budget time for tuning chunking. For production with critical data, it is still better to keep the assistant under supervision, like an intern with access to prod.