Inference

1 · What it is

Inference is the stage where a trained model gets used. You give it an input it has not seen before, and it produces an output: a label for a photo, a number, or a written reply. Training has already set the model’s weights, the learned numbers inside it. During inference those weights are only read, never changed, and they stay loaded in memory while the model serves requests. (In statistics, “inference” means something different, so check which sense a text uses.)

For a simple classifier, one forward pass, a single trip of the input through the model, gives the answer. A language model works in two phases. In prefill, it reads the whole prompt at once, split into tokens of about four English characters each, and stores intermediate results called keys and values. That same pass also yields the first token of the reply. In decode, it adds one token per pass until it hits a limit or an end token. A key-value (KV) cache keeps the stored results so each step does not redo earlier work.

That loop is why speed and cost matter. Generating tokens is almost always the slowest step, so halving the length of a reply can roughly halve the wait, while halving the prompt may save only 1 to 5 per cent. Apps track time to first token and stream text as it is made. Serving large models also needs many GPUs, which makes each request costly.

So engineers choose how and where to run it. Static, or batch, inference computes answers ahead of time, like a weather model that refreshes forecasts every four hours. Dynamic, or online, inference runs on each request, handling any input but needing to be fast. Batching requests raises throughput at some cost to waiting time; Anthropic’s batch API halves the price for jobs that can wait. Quantisation, storing weights with fewer bits, can make a model four times smaller so it runs on a phone, offline, keeping data on the device.