NPUs in laptops: new demands on enterprise IT

The cloud is slowly losing its monopoly on AI computing. Microsoft has made a neural processor a mandatory requirement for Copilot+ PCs, and AMD and Intel are building NPUs directly into their SoCs. Enterprise IT now faces a choice it never had before.

Three processors on one die

Modern SoCs such as the AMD Ryzen AI 300 or Intel Core Ultra contain three compute units: a classic CPU, a GPU and an NPU. That sounds simple on paper, but a neural processor is a very different beast. An NPU is optimized for matrix operations, so neural networks run faster on it than on a general-purpose GPU. But only if the model is smaller than its local memory.

AMD and Intel gave the NPU just 16 GB of VRAM (or less), while the GPU shares the laptop's main system memory. This trade-off shows up in benchmarks. The NPU in the AMD Ryzen AI 300 generated an image in 70 seconds, while the integrated GPU of the same chip finished in 30. The specialized processor lost to the general-purpose one by a factor of two, on the very task it was designed for.

The bottleneck is memory. When a model is larger than the NPU can hold, the neural processor becomes a bottleneck for network artifacts instead of accelerating anything.

A hybrid architecture for enterprise IT

The main scenario promoted by Microsoft and its partners:

• Light AI tasks (classification, search, generating short texts) run locally on the NPU
• Complex models (large LLMs, video processing) run in the cloud
• Data partly stays on the device and is partly synchronized
• Offload logic is built into the application

For enterprise IT, this means new challenges. It used to be possible to roll out a cloud service and forget about it. Now you need to:

Manage models on devices. Each laptop gets a set of ONNX or TensorFlow models. Versions can drift apart. Updates are pulled over the internet. Across a corporation's 50K laptops, this becomes a logistics problem.

Control memory. If a model does not fit into the NPU's 16 GB, the application crashes. Previously, a cloud engineer solved this once. Now you need a client-side algorithm that selects a model based on the device configuration.

Watch the battery. An NPU draws less power than a GPU, but the gain is smaller than it seems at first glance.

What actually works today

Small LLMs (such as Phi-3.5 with 3.8B parameters) fit on a laptop entirely. Inference runs on the NPU in acceptable time. But this works only for reading and classification. Text generation with a full Llama-2-7B is still slow.

Copilot for Microsoft 365 is the main beneficiary. Document search, email summaries and meeting scheduling all run locally. Microsoft promises this will improve privacy (data does not leave for the cloud) and speed (no network delays).

What this means

Laptop makers have overplayed their hand in niche markets. Enterprise IT has received a tool that it must learn to use. The boldest are already preparing model delivery platforms. The rest are waiting for it to get simpler. Nobody is leaving the cloud: a hybrid future, by definition, contains both parts.