Central processing units

1 · What it is

A CPU, short for central processing unit, is the chip that runs a computer’s software. People often call it simply the processor. It carries out instructions, does sums and moves data around.

Cores and threads

A CPU has just a few cores backed by lots of cache memory. Cache is a small, fast store of memory close to the cores. A core is one physical worker inside the chip. A thread is one stream of instructions, like one to-do list, and one core may juggle a single thread or several. A CPU is built to race through steps one after another, and runs only a few dozen threads at once.

Today most CPUs are multi-core: several cores share one chip, so different jobs can run at the same time.

The instruction cycle

Every instruction goes through a repeating loop. Fetch pulls the next instruction out of memory. Decode works out what it means and what it needs. Execute does the operation. Store writes the result to memory or a register, an ultra-fast spot for temporary data.

The control unit directs the other parts. The arithmetic logic unit, or ALU, does sums and logic checks. A clock keeps every step in time. The cache, in levels called L1, L2 and L3, is fast memory near each core that holds often-used instructions and data, so the CPU avoids waiting on slower main memory, a bit like keeping your pencil case on your desk instead of in your locker.

Designers add tricks to cut waiting, such as pipelining, which overlaps the stages of the loop. A higher clock speed alone does not make a faster chip; design, cores, cache and the job matter too.

Where CPUs shine

CPUs are fast and versatile. They suit tasks where you click and type and expect a quick reply, like loading the file you asked for.

Why AI maths is different

AI depends on matrix maths: huge grids of multiplications and additions. A GPU, or graphics processing unit, is a specialist chip that helps the CPU with heavily parallel jobs like graphics and AI. Its hundreds of cores juggle thousands of threads together. Each thread is slower, but far more work gets done.

Picture a grid of sums as a big multiplication table that a model needs to fill in. Each box is one small multiply, and no box depends on its neighbour. A CPU splits the table among its few cores. Even with each core juggling more than one thread, it can run only a few dozen threads at once, so the table gets done a slice at a time, and every other slice waits in line.

A GPU takes a different path. It breaks the job into thousands of separate tasks and works on them at the same moment. Each task is a little slower, but because so many run side by side, the whole table fills in far faster. For about the same price and power, a GPU gets through much more of this kind of work.

The CPU as the organiser

That does not make the CPU useless for AI. The GPU works alongside the CPU; it does not replace it. Arm says the CPU coordinates inference, the work of using a trained model to get answers, and manages the data pipelines, the steps that move data in and out. Like a head chef, it plans the meal while helpers chop.

Sometimes the split changes. When a model is too big for the GPU’s own memory, llama.cpp can put part of it on the GPU and run the rest on the CPU.

CPUs still run AI

The open-source project llama.cpp runs language models on many chips, including Apple silicon and x86 chips with AVX instructions. It can shrink a model with integer quantization, storing numbers with fewer bits, to cut memory use. Google’s LiteRT can use the CPU, the GPU or an NPU, a chip made for AI maths. That lets open models like Gemma run on your own device. ONNX Runtime, a tool for running AI models, can use a GPU when possible and fall back to the CPU.