Gemma 26B fine-tuned on a 24 GB MacBook: offline assistant solves 703 of 766 conversations

Six months of work, one MacBook with 24 GB of unified memory and a hard requirement: no cloud, no subscriptions. The project's author built a voice assistant for macOS that doesn't chat but acts: it opens folders, searches for files, launches movies and reads email. To do this he fine-tuned Gemma 4 26B A4B (a MoE with ~4 billion active parameters) using QLoRA via mlx-lm. The rules that used to take up about 5,000 tokens of system prompt moved into the weights of a small LoRA adapter. On a frozen test bench of 766 conversations, the v4b version with a short prompt solved 703, and the median reply time was 1.64 seconds.

The brain picks a tool instead of writing text

The app is written in Electron, with separate processes running around the window. Python handles hearing, with a speech detector and Whisper built on MLX. The brain is llama-server (llama.cpp) running Gemma in 4-bit quantization with a LoRA in GGUF format. Voice is synthesized by Qwen-TTS from a reference recording. A Swift helper covers macOS system matters (sound, microphone, echo). A watchdog restarts any process that crashes, and the voice comes back up in about 7 seconds.

For each utterance, the model returns one step in a fixed JSON format with the fields think, tool, args, say and next. The app runs the tool and, if needed, passes the result back to the model. Any change or deletion of files requires voice confirmation. ChatGPT, Claude and Gemini produce the same format if the user connects them: tools are passed to them as native function calls. In that case the local llama-server shuts down and frees about 13 GB.

A 766-conversation test bench and the repair pass

Before training the model, the author built a frozen test bench: phrases with reference steps that he never trains on. It covers folder commands, movie search, email, questions about documents, multi-turn chains, refusals and speech-recognition mishearings. Two lessons were expensive. Checking individual turns threw out more than 400 chains and painted too rosy a picture, so whole conversations have to be compared. Examples built from the bench's own errors inflate the score on that same bench, so an honest check needs a fresh set.

Version history (the bench grew from 533 to 766 conversations, so direct comparison is valid only within the same size):

• No adapter, full prompt: 475 of 533 (89%), median 1.94 s
• v3, full prompt: 487 of 533 (91%), median 1.63 s
• v4 with new sections (documents, drawings, English, chains): 684 of 766
• v4b, short prompt: 703 of 766, median 1.64 s

In v4 the model improved on the new sections but regressed on the old ones. "Play a movie" started launching music, "find a movie" lost its search type, and "my own folder" got confused with the notes folder. A repair pass helped: 416 examples covering these cases plus a random sample of old and new data, about 1,100 in total. Further training ran for another ~1000 steps with a learning rate of 1.5e-5 instead of 3e-5 and without validation. It took about 70 minutes and gave the best result of the whole project. Training from scratch, by contrast, made the model forget old material.

What broke on the laptop

The Metal limit dictates example length: no more than ~1400 tokens, or it crashes with Insufficient Memory. Training with the full prompt on a Mac is physically impossible, which is one more argument for moving the rules into the weights. The script discards examples that won't fit in advance. Speed is 0.25–0.4 steps per second, a pass takes about three hours, so the model trains overnight.

Other pitfalls:

• The Mac fell asleep. After one training run, the bench check took six hours. Now the whole chain (training, GGUF conversion, bench) runs under a single caffeinate -i -s.
• Memory failures. The browser and Spotlight indexing push swap to 6–7 GB, and the crashes coincided with validation every 200–400 steps. The fix: save checkpoints, resume automatically with --resume-adapter-file and turn off validation in later stages.
• Editing a bash script on the fly crashed the chain with a syntax error, because bash reads the file as it executes.
• Gemma 4 needed two small patches to mlx-lm, and converting the adapter to GGUF requires a separate environment with torch.

There is plenty of engineering around the model, too. The movie is ducked not on the assistant's name but on the onset of speech: if the voice is 6 dB louder than the "floating floor" and it is not echo, the sound is lowered, and it returns after 2.5 seconds without the name.

A synthesis defect, where "sir" turns into a different vowel in roughly every third take, is caught acoustically. The vowel is measured with 12th-order LPC analysis and the second formant is compared (about 2000–2300 Hz for the correct vowel, 750–900 Hz for the faulty one). The check takes about 3 ms, the detector catches 39 of 46 failures, and the phrase is resynthesized with another take.

Questions about documents are handled by Spotlight: search in 60–700 ms, the top two or three files, windows of ~600 characters and an answer based only on them. DWG drawings are parsed by LibreDWG in WebAssembly in a separate thread, and the result is converted to SVG and cached.

A recipe for your own project

If you are building a local agent, start with a test bench, not with training. Freeze a set of conversations and compare versions only on it, and for the final evaluation keep a fresh sample that neither the model nor you has seen.

Rules that drag thousands of prompt tokens into every request are worth moving into a LoRA: accuracy and latency improve at the same time. Add new skills on top of the previous adapter using a mix of old and new data, with a lowered learning rate. 24 GB is enough for a 26B MoE model if you cut example length, save checkpoints and keep the machine awake. The author's next step is personal memory and self-learning from phrases the assistant did not understand.