How to Build a RAG Bot on a T4 for Civics Textbooks: Qwen Instead of Web Search

A

Small Model with Context Beats a Large One with Internet Access

As Russia rolls out unified state-approved textbooks, one enthusiast decided to build a helper for them. The RAG bot searches for answers to civics questions not on the internet and not in the model's general weights, but in two uploaded textbooks for 9th and 10th grade.

Why does this matter? In class, you get graded for an answer that matches a specific textbook — not for a philosophically sound answer drawn from general knowledge. A typical large model with internet access produces an explanation that's correct but doesn't match the textbook. That calls for a different architecture: a small model that receives a short, narrow context and doesn't drift into general reasoning. A low generation temperature and a system prompt help keep it close to the source text. A restricted context doesn't strip the model of its base knowledge, but it does push it into search mode for specific material rather than open-ended reasoning on a topic.

The Pipeline: From PDF to Answer

The tech stack is simple. pymupdf4llm parses the PDF page by page, converting it to Markdown while preserving tables and lists. MarkdownHeaderTextSplitter then handles three levels: Chapter (#) → Paragraph (##) → Subsection (###). Each fragment retains metadata: grade, paragraph number, and page number.

A second pass — RecursiveCharacterTextSplitter with chunk_size=1000 and chunk_overlap=150 — cuts long sections into manageable chunks. intfloat/multilingual-e5-large embeddings encode the text, and a FAISS database (faiss-cpu) runs in Colab's memory.

When a user writes "give me the answers to the questions for paragraph 13, grade 10," a regular expression extracts the grade and paragraph number. A function called smart_filter checks whether the metadata matches the query. FAISS returns up to four relevant fragments. If the strict filter finds nothing, the code repeats the search filtered only by grade — context may come from a different paragraph in that case.

The retrieved fragments are assembled into a context block with metadata like [Grade 10 | § 13 | Page 45] and passed to Qwen2.5-3B-Instruct along with a system prompt: answer strictly based on the provided text, and cite the page number at the end. The model runs at a temperature of 0.1. After every answer, torch.cuda.empty_cache() is called to free cached memory.

Infrastructure, Problems, and Plans

Model: Qwen2.5-3B-Instruct (3 billion parameters). Quantization: 4-bit NF4 via bitsandbytes — this is what lets the model fit into the T4's VRAM. Embeddings: intfloat/multilingual-e5-large. Vector database: FAISS (faiss-cpu) in RAM. Infrastructure: Google Colab with a free T4 GPU. Everything runs on a single machine.

Answer formatting currently shows up as raw Markdown (the ** markers are visible). Paragraph metadata gets lost on pages without a heading: if a paragraph starts on page 139 and continues on page 140, fragments from the second page end up without metadata and don't show up in paragraph-specific retrieval. There's no fact-checking: the bot doesn't verify whether a phrase in its answer actually matches the retrieved context. The user's grade level isn't remembered between messages. A file's page number may differ from the one printed in the physical edition.

The author plans to scale this up to history and biology textbooks. For upcoming sprints: fix answer formatting, implement session-state storage, add answer validation before sending responses, and solve the lost-metadata problem on multi-page paragraphs.