Vanishing and exploding gradients

Vanishing and exploding gradients happen when the error signal shrinks toward zero or grows very large as it passes back through many layers, which makes deep networks hard to train.

Also known as gradient clipping.