Local RAG with FastAPI and Ollama: From a Simple LLM Call to a Production Backend

A developer on Habr examines one specific moment: when a single call to a local model stops being enough, and what you need to add to get a real backend system.

When a simple call isn't enough

The first version looks obvious: the frontend sends a question, FastAPI accepts `POST /ask`, the backend calls Ollama and returns a string with the answer. That's enough for a demo. It's not enough for a documentation assistant.

Questions come up immediately: which documents does the answer rely on? Which fragments ended up in the prompt? Why does one request take two seconds and another fifteen? What happens if the index hasn't been updated in three weeks? A plain LLM call can't answer any of these.

What the backend layer adds

The author introduces the components one at a time, explaining the specific problem each one solves:

• request_id — a unique request identifier for correlating logs and debugging a specific failure
• sources — a list of document fragments that made it into the model's context, with the source of each
• timings — a breakdown of time by stage: index search, prompt construction, answer generation
• rebuild index — a separate endpoint for rebuilding the index without restarting the service
• negative tests — checks for behavior with an empty corpus, off-topic questions, and a stale index

The stack is fully local: FastAPI for the HTTP layer, Ollama for generation, and custom embeddings. No external API keys.

The API contract as the system boundary

One of the article's key arguments is the importance of an explicit contract at the response level. When `/ask` returns only a string, the service remains a black box. Once the response includes `request_id`, `sources`, `timings`, and the index status, you can monitor, debug, and improve it.

"I'm not showing how RAG works in general, but the path from a simple

call to a local LLM to a small backend project with an API contract, logging, sources, timings, and honest limitations."

This approach changes how you think about the service: instead of "the model answers somehow," you get "the system behaves predictably."

Honesty about limitations

The author explicitly lists what the project lacks: authorization, caching, support for multi-user load, and CI/CD. That makes the article more useful than most tutorials, where the limitations section is either missing or reduced to a perfunctory sentence.

Negative tests get their own treatment. They check behavior with an empty document corpus, with questions that have no relevant content in the index, and when the index is queried after new files are added without a rebuild. It's a minimal set, but it already shows where exactly the system fails.

What this means

The article fills a specific gap: most RAG material either explains the theory or shows a "hello world" with LangChain. Here you get a working local service with observability tooling that you can use as the basis for a real project.