Underfitting

1 · What it is

A machine learning model is useful only if it makes good predictions on examples it has never seen. Underfitting is the failure that comes before that test, because the model never captures the real pattern in its training data. Its predictions are weak on the training set and on new data alike. Google’s crash course compares an overfit model to a gadget that shines in the lab and flops in real use; an underfit one flops in the lab as well.

A scikit-learn example makes this concrete with 30 noisy points taken from part of a cosine wave. It fits three polynomial curves to those points, with degrees 1, 4 and 15. The degree-1 curve is just a straight line, and it cannot bend to match even the training points. The degree-4 curve lands almost exactly on the hidden function. The degree-15 curve chases the random noise in the sample instead, which is overfitting.

Loss is a number that measures how far a prediction is from the correct answer, and training tries to push it down. A loss curve plots that number as training goes on. An epoch is one complete pass through the training set. Draw one curve for the training set and one for a validation set, which is data held back to check the model during development. Together the two curves tell you which problem you have. With underfitting, both curves flatten early at a high loss. Stanford’s CS231n course notes add that when validation results track training results closely, the model probably lacks capacity. In a good fit, both curves fall to a low value and keep similar shapes. A small gap between them is normal. With overfitting, training loss keeps falling or holds steady while validation loss turns and climbs.

Google’s glossary lists the usual suspects. They are the wrong input features, too few epochs, a learning rate set too low, regularization that is too strong, and a neural network with too few hidden layers. Regularization is any technique that reduces overfitting, often by penalizing complex models. Each cause has a matching remedy. Grow the model by adding layers or parameters. Deep learning practitioners often call the count of learnable parameters a model’s capacity. Feed it features that carry the signal, let training run longer, or turn the regularization rate down. A lower regularization rate weakens the pull toward simple models.

Statisticians call this kind of error high bias. The scikit-learn guide defines an estimator’s bias as its average error across different training sets. A learning curve is a related plot: it shows training and validation scores against the number of training examples, not epochs. If both scores level off at a poor value as the training set grows, the model itself is the bottleneck.

Every fix can overshoot. Complex models usually beat simple ones on training data, yet simple ones often do better on the test set. The TensorFlow guide suggests starting with a small network and growing it until the validation loss stops improving much. Early stopping halts a run before the loss has fully settled, which fights overfitting. It usually leaves training loss higher, so stopping far too soon can leave a model underfit. The overfitting explainer covers that opposite failure in detail.