Language Model on ESP32-S3 Sped Up to 406 Tokens/s — 18x Faster
The ESP32-S3 microcontroller, which costs a couple hundred rubles, can run a tiny 260-thousand-parameter Llama-2, but known projects managed only about 19 tokens per second. The author converted the weights to int8, wrote assembly kernels for the PIE vector instructions, brought in the second core and rewrote the attention code. The result is 406 tokens per second, against 22.5 for the naive single-core version. The generated text was checked and matches the reference output. The takeaway for embedded developers is simple: compress the weights and choose where they live first, then reach for assembly.
AI-processed from Habr AI; edited by Hamidun News
Running a language model on a microcontroller that costs a few hundred rubles is more of a hobbyist stunt, but this write-up is useful to anyone squeezing speed out of constrained hardware. The author took the ESP32-S3 (two Xtensa LX7 cores at 240 MHz, 512 KB of internal SRAM) and the stories260K model from Andrej Karpathy's tinyllamas collection. It is a real Llama-2 with attention, RoPE and SwiGLU, with just 259 328 parameters, trained on the TinyStories children's story dataset. The well-known esp32-llm project by DaveBben reached 19.13 tokens per second on this chip. Here the author got 406 tokens per second, roughly 21 times faster. No overclocking, no external accelerators.
Verify first, then optimize
A bug in assembly does not crash the program. It quietly corrupts the text. That is hard to spot: stories about Lily and a red ball are no masterpiece even when the model works correctly. So the author built a chain of checks: a numpy reference; a firmware simulator with the same quantization and the same order of float operations; a C engine on a PC (bit-for-bit identical to the simulator over 200 steps); Espressif's QEMU emulator; and finally the board itself. On the board, every mode produces the same tokens as the simulator, and the logit divergence is no more than 5·10⁻⁸.
Four steps to speed
int8 and where the weights live. Weights in int8, with one scale per matrix row, shrank the model from 1013 to 274 KB. Quality barely suffers: perplexity rises by 0.0%, and the full pipeline with int8 activations and the KV-cache adds +0.7%. Four bits is already destructive: +24.4% with one scale per row and +14.5% with one scale per 16 weights. Moving to int8 in the same PSRAM sped up generation by 2.5x: the bottleneck was memory reads, not arithmetic. Flash via XIP is almost as fast as PSRAM (53.3 vs 55.3 tok/s), and internal SRAM added another jump to 71.7.
PIE vector instructions. This extension, with 128-bit registers, can multiply 16 pairs of int8 and add the result to an accumulator in a single ee.vmulas.s8.accx instruction. The compiler does not generate it, so the kernel is written in assembly. The model dimension of 64 fits into four registers, so the input vector is loaded only once. The Q, K and V matrices are merged into one, as are W1 and W3: four multiplications per layer instead of seven. Result: 71.7 → 175.2 tok/s.
The second core. The matrices are tiny, and a FreeRTOS semaphore would eat the entire gain. So the second core spins on a volatile counter and sleeps outside of generation. The watchdog timer had to be disabled. Result: 285.5 tok/s, a 1.63x gain. It falls short of 2x because RMSNorm, RoPE, SwiGLU and token selection remain sequential.
Attention. Once the matrices were fast, attention accounted for 72% of the time, a textbook case of Amdahl's law. The author processes the two query heads that share one KV head in a single pass, and splits the computation so that everything fits into the 16 FPU registers (the disassembly showed 81 stack accesses). Average speed over 128 tokens rose to 406.1 tok/s. Converting Q·K to integers gained nothing: 403.6 vs 406.1.
Accounting for every kilobyte
The most unexpected part is memory. The weights (274 KB), activations and KV-cache (about 360 bytes per token) must all fit in SRAM. After ESP-IDF booted, 335 KB remained, and the context fit only 96 tokens. Several changes freed up space: moving FreeRTOS code, the heap and the flash driver from IRAM to flash, a 16 KB instruction cache, small UART/USB buffers, and keeping the RoPE table in flash. That left 363 KB and a context of 160 tokens. The author's conclusion: 260 thousand parameters in int8 is almost the ceiling for this chip's SRAM, and anything larger will have to be read from flash or PSRAM.
What this means in practice
Honestly: a model like this has no real use. It can only write children's stories, and the author admits as much. But the lessons carry over. Quantization and smart weight placement deliver more than clever tricks do. At every step the bottleneck moves (memory, the scalar loop, attention, and now divisions and normalizations). And without bit-for-bit verification, low-level optimization is dangerous. The code, assembly kernels, simulator and benchmark script are on GitHub (nikuson/esp32-llm). The author's next goal is stories15M running straight from flash, a model 60 times larger.
- Nov 18, 2026LLMs on a GPU-less VPS: 4B models sort support tickets with 90% accuracy in 3–5 seconds
- Nov 18, 2026Compressed Bonsai 2 27B hit 17 tokens/s on a 2017 GPU, while Qwen3 8B hit 32
- Nov 18, 2026OpenAI releases GPT-6.2 Astra: second place in the rankings and a 20% lower price
- Nov 18, 2026After a Failure, Agents Roll Back Their Work in Only 47% of Cases, Though They Solve 84% of Tasks
Key Questions Answered
What are the key details on Verify first, then optimize?
A bug in assembly does not crash the program. It quietly corrupts the text. That is hard to spot: stories about Lily and a red ball are no masterpiece even when the model works correctly. So the author built a chain of checks: a numpy reference; a firmware simulator with the same quantization and the same order of float operations; a C engine on a PC (bit-for-bit identical to the simulator over 200 steps); Espressif's QEMU emulator; and finally the board itself.
What are the key details on Four steps to speed?
int8 and where the weights live. Weights in int8, with one scale per matrix row, shrank the model from 1013 to 274 KB. Quality barely suffers: perplexity rises by 0.0%, and the full pipeline with int8 activations and the KV-cache adds +0.7%. Four bits is already destructive: +24.4% with one scale per row and +14.5% with one scale per 16 weights. Moving to int8 in the same PSRAM sped up generation by 2.5x: the bottleneck was memory reads, not arithmetic.
What are the key details on Accounting for every kilobyte?
The most unexpected part is memory. The weights (274 KB), activations and KV-cache (about 360 bytes per token) must all fit in SRAM. After ESP-IDF booted, 335 KB remained, and the context fit only 96 tokens. Several changes freed up space: moving FreeRTOS code, the heap and the flash driver from IRAM to flash, a 16 KB instruction cache, small UART/USB buffers, and keeping the RoPE table in flash. That left 363 KB and a context of 160 tokens.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.