Model serving

1 · What it is

Training a model and using it are two different jobs. Model serving is the work of loading a trained model onto a server, a computer that waits for requests, and giving it a network address. Any app can then send the model a request and get an answer back, the same way it would call a weather or maps service.

A homework-help app, for example, sends a student’s question to an address such as /v1/chat/completions. vLLM, an open-source engine for serving large language models, provides a server that answers at addresses like that. It accepts requests in the same format as OpenAI’s Completions API. So any HTTP client, meaning any program that can send web requests, can talk to it.

TensorFlow Serving does a similar job for TensorFlow models. It is built for production, meaning real apps with real users. It is designed so a team can add new algorithms and run experiments without changing the server design or its APIs.

NVIDIA’s Triton Inference Server is another open-source serving system. It can load models built with many tools, including PyTorch, ONNX and TensorRT. Apps can reach it in two ways: HTTP/REST, the usual web style, or gRPC, another way for programs to send requests. Triton can also run several models at the same time on one server, which it calls concurrent model execution.

The hard part is answering many requests at once, quickly, without wasting expensive hardware. Three ideas do most of the work.

Batching

Triton has a dynamic batcher. It groups requests on the server into one batch. That usually raises throughput, meaning answers per second. You can let the batcher wait a short, set time so more requests can join. That wait is the catch. Triton’s own guide describes it as trading more latency, the delay each user feels, for more throughput. By default its batcher does not wait at all. The guide suggests raising the wait step by step and stopping once the delay goes past your latency budget, the longest wait you are willing to accept.

Picture the homework app on a school night. Three students press “send” within a moment of each other. With no wait, the server may run each question separately. With a short wait, all three questions travel through the model together as one batch. Each student waits slightly longer, but the server finishes more answers per second.

Memory for language models

Language models add a twist. Each request keeps a working memory called the KV cache, short for key-value cache. The vLLM research paper explains that this memory is large for each request and grows and shrinks as the answer is written. If it is managed badly, memory is wasted, and that waste limits how many requests fit in one batch. The paper’s fix, PagedAttention, borrows a memory trick called paging from operating systems. The authors report 2 to 4 times more throughput than earlier systems at the same latency. vLLM also uses a method called continuous batching for incoming requests.

Scaling

The third idea is scaling, meaning running more or fewer copies of the model as demand changes. KServe builds on Kubernetes, a system that runs software across many machines, and adds settings made for running models. It handles load balancing (spreading requests across copies), autoscaling (adding or removing copies automatically), canary rollouts and monitoring. A canary rollout lets a team test a new model version on part of the traffic. KServe’s autoscaler can watch how fast the model is writing answers (token throughput, where a token is a small piece of a word), how long the waiting line gets and how busy the GPUs (the chips that run the model) are. It can shrink to zero copies when nobody is calling. It can also cope with sudden rushes of traffic.

Watching and safety

Serving also means watching. Triton reports metrics such as GPU use, server throughput and server latency, so a team can see when answers slow down. Security needs care too. vLLM’s own docs warn not to rely on its API key option alone to protect the server.

What serving does not do is make the model smarter. It runs whatever model it is given, as fast and as cheaply as it can.