Model parallelism splits one model across several chips, so a model too big for a single chip can still be trained or run.
In plain words
The largest models do not fit in the memory of one GPU. With model parallelism, the model is cut into parts, each part lives on a different device, and the devices work together and combine their results into one model.
A quick example
A team's language model has billions of parameters, far more than one GPU can hold. They split each layer across eight GPUs. Every GPU computes its share of each layer and they exchange results, so together they train the model none could hold alone.
Where you'll meet it
- Training very large language models.
- Serving models that do not fit on one GPU.
Related
Sources
- Docsmodel parallelism, Google Machine Learning Glossary · read 29 Sept 2026
- PaperMegatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism, Shoeybi et al. (arXiv) · read 29 Sept 2026
Checked by WhatIsAI against the sources above · 29 Sept 2026. Spotted an error? Report it.