Qdrant runs embeddings on an ESP32 microcontroller: a phrase becomes a vector in 1.2 ms
For a device to search by meaning without the cloud, text must first be turned into a vector of numbers (an embedding). Ordinary models for this weigh anywhere from 310 MB to 4.28 GB. The ESP32 board has only 16 MB of flash memory and no OS, so Qdrant took a static model of 7.5 MB, essentially a table of ready-made vectors for words. A phrase of 10–12 tokens (words and their parts) becomes a vector in 1.2 ms. For microcontrollers, use static models rather than compressed transformers.
AI-processed from Qdrant Blog; edited by Hamidun News
Imagine a smart speaker or a factory sensor that searches by meaning right on the device, without the cloud. To do this, any text is first converted into an embedding: a set of numbers that lets you compare phrases by meaning. Ordinary models for this task are huge.
The Qdrant team showed how to get around this limitation on an ESP32 board. It is a microcontroller with 16 MB of flash memory and 8 MB of PSRAM (fast RAM), and it has neither a disk nor an operating system (OS). A phrase of 10–12 tokens, meaning words and their parts, becomes a vector in 1.
2 ms. That is about 830 phrases per second. The secret is a static model: it is simply a table of ready-made vectors.
Why ordinary models don't work here
The problem is twofold. First, size. A small modern embedding model in 8-bit form takes 310 MB of memory, and a larger one requires 4.28 GB. Against 16 MB of flash on the board, this is not a matter of optimization but a complete incompatibility. Second, computation: transformers are built on heavy matrix multiplications and often drag in bulky dependencies such as PyTorch.
On devices with an OS, several cores and gigabytes of memory, all of this is tolerable. On a board with a couple of megabytes of flash, no disk and no operating system, it is not. The authors put it plainly: a regular embedding model cannot be run on bare hardware.
A static model as a table of words
The way out lies in static embedding models. They are as simple as it gets: every token in the vocabulary has its own vector written in advance. To get the embedding of a phrase, you split the text into tokens, pull the row for each one from the table, average them and, if needed, normalize the result. There are no neural network layers and no matrix multiplications.
A suitable candidate exists among ready-made models: potion-base-2M takes only 7.5 MB, so it fits comfortably into the board's 16 MB of flash. The remaining question is how to get the model and the vocabulary onto the chip when there is nowhere to read files from.
How it all fits into the firmware
The code is written in Rust in no_std mode: it talks to the hardware directly and does not rely on OS services. This forced the team to solve three problems.
The first is the weights. A separate build script on a laptop reads the model file in safetensors format, finds the embedding table and generates constants: the vocabulary size and where the weight bytes begin and end. The model is then embedded in its entirety into the firmware binary. If the file looks wrong, the error surfaces at build time, not on the device. On the chip, the weights are simply read from memory.
The second is the tokenizer. The Hugging Face tokenizer library cannot be used, because it requires the standard library. So the authors implemented the WordPiece algorithm themselves. It greedily searches for the longest fragment of a word that is in the vocabulary, cuts it off and repeats the same with the remainder. Together with text normalization (removing control characters and accents, handling Chinese characters, lowercasing) and splitting on spaces and punctuation, this comes to just over 200 lines of code.
The third is the vocabulary. It is usually a hash table, but the board does not have one. So at build time the vocabulary is packed into a compact data block: first the number of entries, then an index of fixed-size slots with offsets, then the entries themselves in sorted order. Each entry contains a 4-byte identifier, a 1-byte length and the token bytes. Looking up an identifier by string is done with ordinary binary search. Some operations need a heap, and a separate allocator provides it.
What was achieved and the limitations
A phrase of 10–12 tokens becomes an embedding in roughly 1.2 ms, which is about 830 embeddings per second. On a MacBook Pro the same approach gives 11,000–15,000 embeddings per second, although on noticeably longer texts. The chip, on the other hand, needs about 3,000 times less memory.
The limitations are obvious. A static model does not understand context the way a transformer does: the vector is built by averaging the contributions of individual tokens. For short queries, commands and simple semantic search over a small database this may be enough, but for subtle tasks such as distinguishing phrasings that are close in meaning, it is unlikely to suffice. Speed is not record-breaking either, although for devices with infrequent queries this is not critical.
The practical takeaway for embedded developers: if you need embeddings on a microcontroller, do not try to shrink a transformer; start with a static model. It needs only a weight table, a vocabulary and an averaging loop. Qdrant calls this the first half of the task and promises vector search directly on the chip as the next step. The full code is published on the company's blog.
- Nov 21, 2026California Sues OpenAI: Its AI Agents Escaped Control at Least 80 Times
- Nov 21, 2026AURA 2.0 Video Model Starts Talking on Its Own: Response Audio Arrives in 344 ms
- Nov 21, 2026Anthropic raised a record $81.3B in its IPO, but the founders kept control
- Nov 21, 2026ONNX Runtime vanishes from GitHub: Microsoft hides repositories after package hack
Key Questions Answered
Why ordinary models don't work here?
The problem is twofold. First, size. A small modern embedding model in 8-bit form takes 310 MB of memory, and a larger one requires 4.28 GB. Against 16 MB of flash on the board, this is not a matter of optimization but a complete incompatibility. Second, computation: transformers are built on heavy matrix multiplications and often drag in bulky dependencies such as PyTorch. On devices with an OS, several cores and gigabytes of memory, all of this is tolerable.
What are the key details on A static model as a table of words?
The way out lies in static embedding models. They are as simple as it gets: every token in the vocabulary has its own vector written in advance. To get the embedding of a phrase, you split the text into tokens, pull the row for each one from the table, average them and, if needed, normalize the result. There are no neural network layers and no matrix multiplications. A suitable candidate exists among ready-made models: potion-base-2M takes only 7.5 MB, so it fits comfortably into the board's 16 MB of flash.
How it all fits into the firmware?
The code is written in Rust in no_std mode: it talks to the hardware directly and does not rely on OS services. This forced the team to solve three problems. The first is the weights. A separate build script on a laptop reads the model file in safetensors format, finds the embedding table and generates constants: the vocabulary size and where the weight bytes begin and end. The model is then embedded in its entirety into the firmware binary.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.