Tensor parallelism splits the big weight matrices inside each layer of a model across several GPUs, so a layer too large for one chip can still run.
In plain words
A transformer layer is mostly large matrix multiplications. Tensor parallelism cuts each matrix into slices, by column or by row, and gives each GPU one slice. Every GPU computes its part at the same time, then the GPUs exchange results to finish the layer. It needs fast links between GPUs because they talk inside every layer.
A quick example
A model's feed-forward layer is too big for one GPU. With 8-way tensor parallelism, each of 8 GPUs holds one eighth of the first matrix's columns and one eighth of the second matrix's rows. Each computes its share, and a single sum across the GPUs gives the layer's output.
Where you'll meet it
- Training and serving very large transformer models.
- Combined with data and pipeline parallelism in big training runs.
Related
Sources
- PaperMegatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism, Shoeybi et al. (arXiv) · read 29 Sept 2026
- DocsLarge Scale Transformer model training with Tensor Parallel (TP), PyTorch Tutorials · read 29 Sept 2026
Checked by WhatIsAI against the sources above · 29 Sept 2026. Spotted an error? Report it.