Dropout
During training, dropout randomly replaces some input tensor elements with zero.
Dropout changes activations while a module is in training mode. On each forward pass, it independently chooses input values to replace with zero. The next layer therefore sees a sampled thinned network. The original dropout paper describes this as preventing units from co-adapting too much.
Common implementations use inverted dropout. If the drop probability is p, surviving values are multiplied by 1 divided by 1 minus p during training. In PyTorch evaluation mode, the dropout module computes the identity function.
The training and evaluation distinction is operationally important. In training mode, dropout keeps choosing zeroed elements independently on each forward call. It does not replace checking loss on a holdout set, which estimates loss on unseen data.
A large network can fit its training examples too closely.
Compare one layer during training and evaluation.
- 1 · sampleOn each training forward pass, dropout samples an independent binary mask.
- 2 · zeroMasked activation values are replaced with zero.
- 3 · scaleSurviving values are multiplied by 1 divided by 1 minus the dropout probability.
- 4 · evaluateDuring evaluation, the dropout module acts as the identity function.
The random mask changes on each training forward pass; it is not a permanent pruning decision.
| Who | What they ask | What it works with |
|---|---|---|
| Vision model trainer | “Can this hidden layer rely less on one fixed feature combination?” | Activation tensors during each training batch |
| Language model engineer | “Should this residual or feed-forward path be regularized?” | Validation loss across dropout rates |
| Researcher | “Does the model generalize better with dropout enabled?” | Held-out performance from matched training runs |
| Production engineer | “Why do repeated predictions differ in training mode?” | Whether the model is in training or evaluation mode |
- Randomly omitting units reduces opportunities for fixed co-adaptations during training.
- One training run samples many different thinned subnetworks.
- Inverted dropout lets the full network run without random masking at evaluation time.
- It does not replace a holdout set for estimating loss on unseen data.
- Too much dropout can reduce the model's predictive power.
- In training mode, dropout continues to choose values independently for each forward call.
- PyTorch advises using training mode for training and evaluation mode for evaluation.
Sources used
This explainer is written in original language. The links below support its factual claims.
- paperDropout: A Simple Way to Prevent Neural Networks from Overfitting, Srivastava et al., JMLR · read 27 Sept 2026
- docsDropout, PyTorch · read 27 Sept 2026
- paperImproving neural networks by preventing co-adaptation of feature detectors, Hinton et al. · read 27 Sept 2026
- docsMachine Learning Glossary, Google for Developers · read 27 Sept 2026
- docsAutograd mechanics - Evaluation Mode, PyTorch · read 27 Sept 2026