Masked language modeling trains a model by hiding some words in a sentence and asking it to guess them from the words on both sides.
In plain words
Chat models learn by predicting the next word from the words before it. Masked language modeling instead blanks out words anywhere in a sentence and has the model fill them in, so it can use context from the left and the right. That makes it good at understanding whole passages rather than writing new text. BERT is the best-known model trained this way.
A quick example
Give a masked language model the sentence "The Milky Way is a [blank] galaxy." It scores possible fillers and ranks a word such as "spiral" near the top, because both sides of the gap point to it.
Where you'll meet it
- Pretraining encoder models such as BERT.
- Search, classification and other understanding tasks built on those models.
Related
Sources
- DocsMasked language modeling, Hugging Face Transformers docs · read 29 Sept 2026
- PaperBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Devlin et al. (arXiv) · read 29 Sept 2026
- DocsMachine Learning Glossary, Google for Developers · read 29 Sept 2026
Checked by WhatIsAI against the sources above · 29 Sept 2026. Spotted an error? Report it.