DeepSpeed is an open-source library, once a key piece of Microsoft's AI at Scale initiative, that makes training and running very large models across many GPUs practical.
In plain words
Big models can run out of memory on each GPU. DeepSpeed's best-known idea, ZeRO (Zero Redundancy Optimizer), cuts memory per GPU: it splits the optimizer states, gradients and weights across GPUs, and can offload to CPU memory or NVMe drives. It works in stages, each saving more memory, and is switched on through a configuration file rather than rewriting model code.
A quick example
Imagine a team whose model runs out of GPU memory during training. They add a DeepSpeed config turning on ZeRO stage 2, so optimizer states and gradients are split across their GPUs, and training fits without changing the model code.
Where you'll meet it
- Training large models across many GPUs.
- Stretching GPU memory by offloading to CPU memory or NVMe drives.
Related
Sources
- DocsDeepSpeed, DeepSpeed · read 29 Sept 2026
- DocsZero Redundancy Optimizer tutorial, DeepSpeed docs · read 29 Sept 2026
- DocsDeepSpeed README, DeepSpeed (GitHub) · read 29 Sept 2026
Checked by WhatIsAI against the sources above · 29 Sept 2026. Spotted an error? Report it.