Continuous batching is a way of serving language models where finished requests leave the batch and new ones join at every step, instead of waiting for the whole batch.
In plain words
A model writes answers one token at a time, and answers differ in length. With plain batching, short answers finish and then sit idle until the longest one is done. Continuous batching decides the batch again at every step: as soon as one answer ends, a waiting request takes its slot, so the GPU stays busy.
A quick example
Ten users send questions at once. One wants a yes or no, another a long essay. With continuous batching the short reply returns after a few steps and a new user's request slides into its place right away, rather than waiting for the essay to finish.
Where you'll meet it
- Serving engines such as vLLM and Hugging Face text-generation-inference.
- Chat services that must handle many users per GPU.
Related
Sources
- PaperOrca: A Distributed Serving System for Transformer-Based Generative Models, Yu et al. (USENIX OSDI 2022) · read 29 Sept 2026
- DocsHow continuous batching enables 23x throughput in LLM inference while reducing p50 latency, Anyscale · read 29 Sept 2026
- PaperEfficient Memory Management for Large Language Model Serving with PagedAttention, Kwon et al. (arXiv) · read 29 Sept 2026
Checked by WhatIsAI against the sources above · 29 Sept 2026. Spotted an error? Report it.