RoBERTa is a 2019 language model from Facebook AI that showed BERT had been undertrained, and got better results by training the same design longer on more data.
In plain words
RoBERTa did not invent a new architecture. The team re-ran BERT's pre-training and carefully tested choices such as training data size and hyperparameters. With better settings and 160 GB of text, including books, Wikipedia and news, it matched or beat every model published after BERT at the time. Like BERT, it learns by hiding 15 per cent of words and predicting them, and it is mostly used as a base to fine-tune for tasks such as classification.
A quick example
Imagine a team building a spam filter. Rather than training from scratch, they take the pre-trained roberta-base model and fine-tune it on a few thousand emails labelled spam or not spam, turning it into a classifier.
Where you'll meet it
- Text classification, named entity recognition and question answering.
- A strong encoder base for fine-tuning.
Related
Sources
- PaperRoBERTa: A Robustly Optimized BERT Pretraining Approach, Liu et al. (arXiv) · read 29 Sept 2026
- DocsFacebookAI/roberta-base, Hugging Face · read 29 Sept 2026
Checked by WhatIsAI against the sources above · 29 Sept 2026. Spotted an error? Report it.