Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseInsight · On RadarHabr AI→ original

After a Claude Ban, a Developer Saved a Year of Chats: 146 MB Turned into a SQLite Database

Anthropic blocked the author's Claude account (he blames a VPN), his $100 subscription was lost, but he was still allowed to export his 146 MB chat archive. That's roughly 46 million characters, which won't fit into a chat context window. So he split the archive into about 12,000 chunks and stored them in SQLite (a lightweight single-file database), and the agent pulls out a few relevant pieces on demand. This is RAG. For a personal archive of this size, a separate vector database isn't necessary.

AI-processed from Habr AI; edited by Hamidun News
After a Claude Ban, a Developer Saved a Year of Chats: 146 MB Turned into a SQLite Database
Source: Habr AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
Search is hybridkeyword-based (FTS5, BM25) and semantic, merged with RRF using K = 10
Key metric #2Markdown files in Obsidian are the source of truth; the SQLite index can be rebuilt at any time
Key metric #3The agent accesses the database through an MCP server, with rules defined in AGENTS.md

Imagine spending several hours a day for more than a year working with an AI assistant: discussing projects, making decisions, and writing down why you chose one option over another. Then one day your account gets blocked. That's what happened to the author of a post on Habr: Anthropic cut off his access to Claude, which he believes was due to his country and his use of a VPN.

The $100 subscription was lost, but the loss of his history worried him more than the money. Fortunately, he was still allowed to export the 146 MB chat archive (about 46 million characters of text). He then built a RAG system.

This is a setup in which the AI doesn't read everything. Instead, it finds several relevant fragments in your database based on the question and answers using them. The database is stored in SQLite, a lightweight single-file database, and contains about 12,000 fragments.

Why the archive can't simply be attached to a chat

The export included Claude.ai conversations, local Claude Code sessions, memory, projects, and artifacts. The conversations.json file alone was about 146 MB, and Cyrillic text was written as escape sequences such as \u041f. Such a file can't be uploaded into a new chat because the context window is limited. Even if it could, it wouldn't solve the problem. The author needs to ask about a specific project or decision and get a few relevant pieces, not carry a year of dialogue around.

He didn't use subagents to process the archive. The orchestrating agent sometimes loses track of its workers, the workers write in different styles, and merging their results consumes even more tokens. Instead, he wrote ordinary deterministic scripts. They migrate conversations, memory, and artifacts and build INDEX.md. Project names are normalized through an alias table, because one project might have been named differently across chats. The script reads the source once, so the model doesn't spend tokens on millions of characters. After the first migration, the author fixed the structure by hand, and the scripts don't overwrite those changes.

Markdown as the original, SQLite as the index

The knowledge base lives in Obsidian as plain files: Memory (profile and preferences), Projects, Topics, and Artifacts. The rule is simple: Markdown is the source of truth, and SQLite is an index that can be deleted and rebuilt. If the author edits a file in Obsidian, that edit is considered correct.

He didn't set up a separate vector database. For 12,000 chunks (the small pieces of text that documents are split into for search), SQLite is enough. It stores files, fragments, an FTS5 full-text index, and embeddings (numerical "fingerprints of meaning") as binary float32 arrays. Similarity is computed with numpy by brute-force comparison. If the data grows by orders of magnitude or measurements reveal a bottleneck, the architecture can be replaced.

Chunking follows headings and paragraphs so that a thought isn't cut in the middle. The target chunk size is about 600 tokens, with a maximum of 1,000. A new heading doesn't close a chunk while that chunk has fewer than 200 tokens. Each chunk keeps its heading and line range, so search results point to an address such as "file.md, lines 42–77."

Content is split into two tiers. The first tier includes memory, notes, and overview files (about 468 chunks). The second tier contains conversation history (more than 11,000 chunks). The first tier has a ranking multiplier so that old dialogues don't push out the profile and CONTEXT.md.

Search is hybrid: keyword-based (FTS5 and BM25) and semantic (text-embedding-3-small, 1536 dimensions). The two result lists are merged with Reciprocal Rank Fusion, so a fragment that ranks high in both lists rises to the top. The author chose K = 10, because with a larger value the gap between top and middle positions became too flat.

MCP and operational lessons

Access to the database is provided by an MCP server over stdio using JSON-RPC 2.0. The model has tools for search, reading line ranges, loading project context, writing, appending, saving a decision note, checking status, listing projects, and reindexing. A read returns the file's hash, and a write that includes that hash reports a conflict if the file has been changed in the meantime.

The first test revealed a weak spot. When asked "what's my name?", the agent answered without checking the database, guessing the name from indirect clues in the system. A session-start rule in the global AGENTS.md and a separate knowledge-first skill, loaded on a trigger, fixed this. Connecting a tool isn't enough; the model has to be made to use it.

The index is synced with the disk no more than once every 10 seconds, and scanning several hundred files takes about 0.03 seconds. If the embeddings API is unavailable, a fresh note can still be found by keywords, and its vector is computed later.

Another lesson concerned diagnostics. Requests to the embedding provider were hanging, while curl returned truncated responses. It turned out that curl on macOS went direct, while Python used the system proxy. From the Python runtime, six chunks received embeddings in four seconds. Always test the network path the application actually uses.

What follows

This is a personal project on a single corpus, not a universal recipe. At larger volumes, the choice of SQLite will have to be reconsidered. The practical takeaway for developers: export your data from AI services regularly, and keep your knowledge in plain files. For a small RAG setup, SQLite, hybrid search, and a few clear rules for the agent are often enough.

Story

Key Questions Answered

Why the archive can't simply be attached to a chat?

The export included Claude.ai conversations, local Claude Code sessions, memory, projects, and artifacts. The conversations.json file alone was about 146 MB, and Cyrillic text was written as escape sequences such as \u041f. Such a file can't be uploaded into a new chat because the context window is limited. Even if it could, it wouldn't solve the problem. The author needs to ask about a specific project or decision and get a few relevant pieces, not carry a year of dialogue around.

What are the key details on Markdown as the original, SQLite as the index?

The knowledge base lives in Obsidian as plain files: Memory (profile and preferences), Projects, Topics, and Artifacts. The rule is simple: Markdown is the source of truth, and SQLite is an index that can be deleted and rebuilt. If the author edits a file in Obsidian, that edit is considered correct. He didn't set up a separate vector database. For 12,000 chunks (the small pieces of text that documents are split into for search), SQLite is enough.

What are the key details on MCP and operational lessons?

Access to the database is provided by an MCP server over stdio using JSON-RPC 2.0. The model has tools for search, reading line ranges, loading project context, writing, appending, saving a decision note, checking status, listing projects, and reindexing. A read returns the file's hash, and a write that includes that hash reports a conflict if the file has been changed in the meantime. The first test revealed a weak spot.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Hands-On AI Engineering · Hamidun Academy

Build & Ship Production AI Agents Without Boilerplate Code

Start your free 72-hour trial at Hamidun Academy (academy.hamidun.com) — interactive lessons on Claude Code & Cursor, live Tech Radar, and production agent templates.

Start 72h Free Trial →Vibe-Coding Intensive
What do you think?
Loading comments…