Concepts
Tensor parallelism
Reference entryTensor parallelism
Tensor parallelism splits individual tensor operations and parameter matrices across devices, allowing them to collaborate within the same model layer.
Where it sits
Tensor parallelism splits individual tensor operations and parameter matrices across devices, allowing them to collaborate within the same model layer.