Neural processing units

1 · What it is

A neural processing unit, or NPU, is a part of a chip made for one job: running the maths inside AI models. Your phone or laptop also has a CPU, the main all-purpose processor, and a GPU, the graphics chip. The NPU is the specialist next to them. Qualcomm says its NPU is a little harder to program, but in return it is built for top performance and low power.

Why a special chip? The operations inside AI are the multiplications, additions and other small steps that AI needs in huge numbers. Microsoft says the NPU works on lots of data in parallel, doing trillions of these steps per second. Microsoft adds that for AI jobs it wastes less energy than a CPU or GPU would, so the battery lasts longer.

Now follow one example from start to finish. Picture an app that writes captions for what people say on a video call. The app is made up, but it follows the steps Microsoft describes for Windows. The captions run all call long, so battery life matters.

Step 1 is to convert the model so it fits the NPU. A model is trained (taught from examples), and models are usually trained with numbers in a large format called FP32. Many NPUs can only do maths on smaller whole numbers, such as INT8, because that is faster and saves power. So the model has to be converted, which is called quantizing. Many models are already available in a converted, ready-to-use form, so this step is often done for you.

Step 2 is to pick the right chip. Microsoft says the recommended way to run AI on the NPU in Windows is a tool called Windows ML. Windows ML looks for the chips the computer has and picks the fastest option it finds. If that option fails or is missing, it falls back to the GPU or CPU.

Step 3 is the NPU doing the maths. Intel says its NPU has special blocks for AI steps such as multiplying big grids of numbers. Intel’s NPU keeps its work in small, fast memory on the chip itself, and moves data to the computer’s main memory as little as possible, so it does more work for each bit of power.

Step 4 is the caption text going back to the app. The work happened on the laptop, not on a server. Keeping the work on the device also helps privacy, because the data stays there.

Where will you find NPUs? Apple’s first Neural Engine shipped in the A11 chip of the iPhone X in 2017. Face ID (unlocking the phone with your face) and Memoji (animated emoji that copy your face) ran on it, right on the phone. Intel puts an NPU inside its Core Ultra processors. Copilot+ PCs are a newer kind of Windows 11 computer, each built around a fast NPU. That NPU is rated above 40 trillion operations per second, written as TOPS.

What does a TOPS number mean for you? So a rating of “40 TOPS” means the chip can do over 40 trillion of those steps each second. Many of the newest Windows AI tools only work with an NPU that fast.

The NPU does not replace the CPU and GPU; the three chips share the work. When the Neural Engine takes the heaviest AI work, the CPU and GPU are left free for other jobs. An app that ignores the NPU gets none of that benefit.