Support vector machines

1 · What it is

A support vector machine, or SVM, is a supervised learning method best known for sorting examples into classes, though it can also do regression and outlier detection. The approach came out of computer science research in the 1990s. Its standard form is usually traced to papers by Boser and colleagues in 1992 and by Cortes and Vapnik in 1995.

Picture each training example as a point, with one axis per feature. A new example gets the class of whichever side of the boundary it lands on. With two features that boundary is a straight line, and with more features it is a flat surface called a hyperplane. Many lines may separate the classes. The SVM keeps the one with the widest margin: the gap between the line and the closest training points from either class. As a rule, the wider that gap, the better the model tends to do on examples it has never seen. The points sitting on the edges of that margin are the support vectors. Moving any other point leaves the line where it is, as long as the point does not cross into the margin.

Real data rarely splits perfectly, so practical SVMs use a soft margin. Points may sit inside the margin or on the wrong side of the line, and each one adds a penalty. That penalty is called the hinge loss, and its aim is a boundary that stays well clear of the training points on both sides. Points safely on the right side of the margin add nothing, so they do not shape the result. In scikit-learn, C is the dial between two goals: few training mistakes and a simple boundary. Set C low and the model puts up with some mistakes in return for a smooth boundary. Set it high and it pushes to get every training point right. Some textbooks define C the other way round, as a budget for how much the margin may be violated, so check which convention a source uses.

For curved borders, SVMs turn to a kernel, a function that scores how similar two examples are. Using one lets the SVM work as if the data had been mapped into a much larger feature space, without ever building that space. The boundary is straight in that larger space, which shows up as a curve in the original features. Google’s glossary gives an example of a hundred input features mapped internally into a million dimensions. The popular RBF kernel, also called the radial kernel, works by distance, so only nearby training points sway a prediction. Its gamma setting sets that reach: raise gamma and each example only sways points very near it. The LIBSVM guide suggests starting with RBF. It recommends picking C and gamma with a grid search scored by cross-validation. Scikit-learn’s SVC class is built on LIBSVM. That library covers classification, regression and one-class SVMs.

Support vector regression, or SVR, applies the same idea to predicting numbers. Small misses are free: a training point whose predicted value lands near the true number adds no cost. When a straight boundary is enough, scikit-learn’s LinearSVC copes with millions of examples, and its training time grows roughly in step with the data. Logistic regression is often the better pick when the classes blur into each other; an SVM usually has the edge when a clear gap splits them.