Pipeline parallelism splits a model's layers into stages on different GPUs and streams small slices of each batch through them like an assembly line.
In plain words
If a model is too big for one GPU, you can put the first group of layers on GPU 1, the next on GPU 2, and so on. Pipelining cuts each batch into micro-batches, so while GPU 2 handles the first slice, GPU 1 can already start the second. The GPUs pass activations forward and gradients backward.
A quick example
A 48-layer model is split into four stages of 12 layers, one per GPU. A batch is cut into eight micro-batches. Once the pipeline fills, all four GPUs are busy at once, each working on a different micro-batch.
Where you'll meet it
- Training models too large for one GPU.
- Large runs that mix pipeline, tensor and data parallelism.
Related
Sources
- PaperGPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism, Huang et al. (arXiv) · read 29 Sept 2026
- DocsPipeline Parallelism, DeepSpeed documentation · read 29 Sept 2026
Checked by WhatIsAI against the sources above · 29 Sept 2026. Spotted an error? Report it.