72B Model on a MacBook M2 Pro Managed 1–2 Tokens per Second and Overheated the Laptop

A typical story: you read a couple of articles, type ollama pull in the terminal, and suddenly you have "your own AI agent that knows all your code." QA engineer Egor Laptev from SENSE tested this scenario on a MacBook Pro M2 Pro with 32 GB of memory and got a textbook lesson in what tutorials don't tell you.

What went wrong

He took Qwen Coder 72B and compressed it to Q2_K, about 26 GB. Technically, the model fit into 32 GB. In practice, Ollama grabbed almost all the memory, macOS went into heavy swapping, the SSD whirred, and the CPU and GPU sat at 100%. Because Apple Silicon uses unified memory, the load lands on the whole system at once. The browser lagged, the IDE froze, and after a few minutes thermal throttling kicked in.

Limiting memory didn't help. The standard advice about 4 GB doesn't work: 26 GB doesn't fit within it, and Ollama simply returns an error. He had to raise the limit to 28 GB. The model ran, but at 1–2 tokens per second. A simple request like "write a JSON parser" takes a minute or two. The temperature dropped from 95°C+ to 85–90°C, making the laptop usable for work, but not comfortable.

There is also a hidden cost. Constant swapping wears out the SSD, so this kind of experiment shouldn't be run for hours. The author, however, reports that the laptop is intact: throttling was a protective reaction, not a failure.

The desktop helped, but not for free

On a PC with an RTX 4070 Ti Super (16 GB VRAM), 32 GB DDR5 and an i5-13600KF, the same model produced 5–8 tokens per second. A reply to a simple request took 10–20 seconds. Part of the weights still lived in system RAM, and it showed. The main downside is that the graphics card is 80–90% occupied while the agent runs. You can only game or edit video after stopping the agent.

RAG: where the bill grows

A model without context is not yet an agent. For it to know your code, docs and tickets, you need RAG: a vector database, embeddings, search. The author estimated the volumes. A personal project of 14 thousand chunks takes about 200 MB. Internal docs of 1.4 million chunks yield about 20 GB. A full domain of 14 million chunks comes to about 200 GB. An HNSW index adds 30–50% overhead. And if you also fine-tune the model, optimizer states multiply memory by roughly 3–4 times.

The hardware bottom line: a workstation with an RTX 4090 or A5000, 64 GB RAM and a 4 TB SSD costs about $6500. Electricity adds $100–180 per year. The most expensive part, though, is that a personal device turns into a server.

What to do about it

The author's conclusion is quite sober. Locally, it makes sense to keep 7B–14B models in Q4_K_M or Q8: by his estimate, that is about 90% of the quality at a third of the resources. They suit quick tasks and private code. Heavy reasoning, RAG over hundreds of gigabytes and fine-tuning are better handed off to the cloud or a GPU cluster.

The limitations should be stated too. This is a single experience on specific hardware, not a benchmark. Q2_K was chosen deliberately as a stress test, and the author himself admits that the quality of such a model is low. The storage and cost figures are approximate.

The practical lesson is simple: before pulling a 72B model onto a laptop, calculate the memory for weights, context and index. If even one of them falls short, take a smaller model or the cloud. And don't trust tutorials that promise "it's all easy."