Vanishing and exploding gradients are training failures in deep networks where the learning signal shrinks to almost nothing or grows out of control as it passes through layers.
In plain words
A network learns by sending an error signal, the gradient, backwards through its layers. In deep networks and recurrent networks that signal is multiplied many times. If it keeps shrinking, early layers barely change and stop learning: vanishing gradients. If it keeps growing, weight updates become huge and training breaks: exploding gradients. Fixes include gradient clipping, which caps the signal, LSTM cells, and residual connections.
A quick example
Imagine training a recurrent network on long sentences and seeing the loss suddenly jump to infinity. Adding gradient clipping, which limits how big any update can be, is a common first fix.
Where you'll meet it
- Training recurrent networks on long sequences.
- Very deep networks, which is one reason residual connections exist.
Related
Sources
- DocsMachine Learning Glossary, Google for Developers · read 29 Sept 2026
- PaperOn the difficulty of training Recurrent Neural Networks, Pascanu, Mikolov and Bengio (arXiv) · read 29 Sept 2026
Checked by WhatIsAI against the sources above · 29 Sept 2026. Spotted an error? Report it.