Menu
Concepts

Regularization

Explainer5 min readBeginnerUpdated 1 Oct 2026
1 · In one line

Regularization is any change to training that limits how closely a model can fit its training examples, so it does better on new data.

1 · What it is

Training adjusts a model’s weights, step by step, until its predictions fit the training examples. Push that too far and the model starts copying accidents of the data instead of the general pattern, which is overfitting. Regularization covers every trick that makes that copying harder. Its goal is lower error on data the model has never seen, even when error on the training set goes up a little. A simpler rule often carries over to new cases better than a complicated one, even if it looks slightly worse in training.

Weight penalties add a cost for big weights to the loss. L2, the version used in ridge regression, pulls large weights toward zero but leaves them slightly above it; with plain stochastic gradient descent, it is the same as weight decay. In code this is often one setting: PyTorch’s AdamW optimizer applies a weight decay of 0.01 unless you change it. L1, the version behind the lasso, charges for each weight’s absolute size and can push some weights all the way to zero, dropping those features. Dropout switches off a random share of a network’s units at every training step, so no unit can lean on one particular partner. PyTorch’s dropout layer zeroes each value with a default probability of 0.5 and scales up the values that remain. Each step therefore trains a different thinned network, and the full network used afterwards acts like an average of all of them. Early stopping checks validation loss as training runs and halts once it has stopped improving. A patience setting lets it wait a few epochs first, and it can roll back to the best weights it saw. Data augmentation creates extra training examples by flipping, rotating or cropping the ones you already have, so the model meets more variety. Dropout and augmentation both switch off once training ends.

None of this is free. Regularization usually costs a little training accuracy in exchange for better results on new data. Turn any of them up too far and the model becomes too simple to learn the real pattern, which is underfitting. So the strength, often written λ, is a setting you choose by comparing several values on validation data.

2 · Why it exists

A model trained only to cut its training error will learn the noise along with the pattern.

Memorized examplesA flexible model can match its training data so closely that it gets new cases wrong.
Fragile teamworkInside a large neural network, units can come to rely on particular partners instead of learning features that are useful on their own.
Training too longIf training keeps going, error on held-out validation data can start rising even while training error keeps falling.
3 · How it works

Follow one training step with an L2 weight penalty added.

An L2 penalty shrinks big weights and narrows the gap between training and validation error One training step: training loss, then the highlighted step that adds lambda times the sum of squared weights, then an update that shrinks big weights, giving a simpler rule. Below, illustrative training and validation errors for lambda at zero, lambda tuned and lambda too large. ONE TRAINING STEP WITH AN L2 PENALTY (ILLUSTRATIVE NUMBERS) 01 · TRAINING LOSS How wrong it is loss = 0.80 on a training batch 02 · KEY STEP: PENALTY Charge big weights + λ × Σ w² w = 0.2, −0.5, 5.0 03 · UPDATE Lower the total loss + penalty big weights shrink 04 · RESULT Smaller weights simpler rule fits less noise Σ w² = 25.29, and 5.0² = 25 of it TRAIN VS VALIDATION ERROR AS λ GROWS tuned λ λ = 0 · NO PENALTY Overfits train 2% validation 18% gap 16 points learned the training noise λ TUNED ON VALIDATION Generalizes train 5% validation 9% gap 4 points best on new data λ TOO LARGE Underfits train 20% validation 22% both errors high too simple for the pattern
Illustrative numbers. The penalty makes big weights expensive, which trades a little training error for a smaller gap on new data, until too much penalty makes the model underfit.
  1. 1 · measureWork out the ordinary loss, which says how wrong the model is on a batch of training examples.
  2. 2 · penalizeAdd a penalty that grows with the weights; for L2 it is λ times the sum of every weight squared.
  3. 3 · updateThe optimizer lowers loss and penalty together, so a big weight survives only if it cuts the loss enough to pay its cost.
  4. 4 · tuneTry several values of λ and keep the one that gives the lowest error on held-out validation data.

λ sets the price of complexity: at zero there is no regularization at all, and set too high it starves the model of the detail it needs.

4 · Where it's used
WhoWhat they askWhat it works with
Spam-filter team“Why does our filter catch almost every old spam email but miss new ones?”Validation error at several L2 strengths
Photo classifier trainer“Will flipped, rotated and cropped copies of our images help?”Validation accuracy with and without data augmentation
Neural-network trainer“How much dropout should this layer use, and when should training stop?”Validation loss per epoch for several dropout rates
Analyst with hundreds of inputs“Which features does the model actually need?”Coefficients that an L1 penalty leaves above zero
5 · What it solves, and what it doesn't
solves
  • Turning up regularization usually raises training loss a little, yet usually improves predictions on real-world examples.
  • L1 tends to leave fewer non-zero weights, so the model ends up depending on fewer features.
  • Dropout has made neural networks better at image, speech, text-sorting and biology tasks.
  • Early stopping is easy to bolt onto training without disturbing how the model learns.
doesn't solve
  • Too much regularization makes a model underfit, too simple to capture the pattern it was meant to learn.
  • The best strength depends on the data, so there is no universal setting and every project must tune its own.
  • Early stopping is quick, but a model trained with a well-chosen penalty is usually better.
  • With adaptive optimizers such as Adam, an L2 penalty and weight decay stop being the same thing, so a setting's name can mislead.
6 · Go deeper
OverfittingUnderfittingBias–variance tradeoffDropoutData augmentation

Sources used

This explainer is written in original language. The links below support its factual claims.

  1. docsMachine Learning Glossary, Google for Developers · read 1 Oct 2026
  2. docsOverfitting: Model complexity, Google for Developers · read 1 Oct 2026
  3. docsOverfitting: L2 regularization, Google for Developers · read 1 Oct 2026
  4. paperDropout: A Simple Way to Prevent Neural Networks from Overfitting, Journal of Machine Learning Research · read 1 Oct 2026
  5. paperDeep Learning, Chapter 7: Regularization for Deep Learning, Goodfellow, Bengio and Courville (MIT Press) · read 1 Oct 2026
  6. paperDecoupled Weight Decay Regularization, arXiv · read 1 Oct 2026
  7. docsDropout, PyTorch · read 1 Oct 2026
  8. docsEarlyStopping, Keras · read 1 Oct 2026
  9. docs1.1. Linear Models, scikit-learn · read 1 Oct 2026
  10. docsData augmentation, TensorFlow · read 1 Oct 2026
  11. docsAdamW, PyTorch · read 1 Oct 2026
Report an error