Naive Bayes

1 · What it is

Naive Bayes is a family of classifiers built on Bayes’ theorem. A classifier looks at one example, such as an email, picks out useful features, and puts it into one of a fixed set of classes, such as spam or not spam. Bayes’ theorem lets you turn a probability around: from how likely the words are in spam to how likely the email is spam given its words. The method goes back to 1961, when it was invented to sort documents into subject categories.

The naive part is one bold shortcut: once you know the class, each feature tells you nothing about the others. For text that is plainly false, since words such as “hong” and “kong” tend to appear together. The usual text version also ignores word order and treats each document as a bag of words. In return, each word’s numbers can be learned on their own, which keeps training simple when there are many words.

Training is counting. The prior is the share of training examples in each class. A word’s likelihood for a class is how often it appears among all the words in that class’s training documents. To score a new email, the model multiplies the prior by the likelihood of each word and picks the class with the biggest product. Real systems add logarithms instead of multiplying, because multiplying many tiny numbers can underflow. One trap remains: a word never seen with a class gives a likelihood of zero, and that zero cancels all the other evidence. Add-one, or Laplace, smoothing fixes this by adding 1 to every count before dividing. In scikit-learn, the amount added is a setting called alpha, and MultinomialNB sets it to 1 by default.

Variants differ in how they model each feature. Multinomial naive Bayes uses word counts and is a classic choice for text. Bernoulli naive Bayes only records whether each word is present, and counts a missing word as evidence too. Gaussian naive Bayes handles measurements, assuming each follows a bell curve within a class. A 1998 comparison found Bernoulli sometimes better with small vocabularies and multinomial usually better with large ones.

Its probability estimates are often poor, yet it still classifies well, because picking a class only needs the right class to score highest. One study argues that dependencies between features can spread evenly across the classes or cancel each other out. The scikit-learn guide calls it extremely fast next to more complex methods. It also warns that its probability outputs are not to be taken too seriously.

Spam filtering was one of the first uses of naive Bayes for text. A 1998 junk-mail study built its filter with a naive Bayesian classifier. It marked a message as junk only when the model gave more than a 99.9% chance. The authors did not trust the model’s probabilities to be accurate, but found that threshold still reasonable.