A learning-rate schedule changes how big the training steps are as training goes on, for example starting gently, then shrinking steps towards the end.
In plain words
The learning rate sets how far each training step moves the weights. One fixed value is rarely best. A common pattern is warmup, a gentler start that helps avoid problems early in training, followed by decay, where the rate is lowered as training goes on. Some schedules restart the rate periodically, and some cut it when validation progress stalls.
A quick example
In PyTorch you create an optimizer, then wrap it in a scheduler such as CosineAnnealingLR. After each optimizer step you call scheduler.step(), and the learning rate follows the chosen curve.
Where you'll meet it
- Pretraining and fine-tuning large models.
- Training with very large batches, where warmup helps.
Related
Sources
- Docstorch.optim: how to adjust learning rate, PyTorch docs · read 29 Sept 2026
- PaperAccurate, Large Minibatch SGD: Training ImageNet in 1 Hour, Goyal et al. (arXiv) · read 29 Sept 2026
- PaperSGDR: Stochastic Gradient Descent with Warm Restarts, Loshchilov and Hutter (arXiv) · read 29 Sept 2026
- DocsMachine Learning Glossary, Google for Developers · read 29 Sept 2026
Checked by WhatIsAI against the sources above · 29 Sept 2026. Spotted an error? Report it.