What are Decoder-only models?

1 · In one line

A decoder-only model predicts each next token from the tokens that came before it.

1 · What it is

The first GPT used a multi-layer Transformer decoder for language modeling. It used masked self-attention heads. That mask keeps a position from reading tokens to its right.

GPT-2 was trained to predict the next word from previous words.

GPT-3 is an autoregressive language model. PaLM is a densely activated Transformer language model.

2 · Why it exists

A causal language model can attend only to tokens on the left.

Future leakageA causal language model can attend only to tokens on the left.

Repeated choiceGPT-2 predicts the next word from all previous words.

Shared interfaceA prompt and demonstrations can specify a task through text interaction.

3 · How it works

Follow a prompt through one next-token step.

The model can attend only to tokens on the left. It predicts the next token in the sequence.

1 · prefixThe model receives the tokens generated or supplied so far.
2 · maskMasked self-attention blocks information from subsequent positions.
3 · scoreThe final decoder state produces a distribution over target tokens.
4 · nextCausal language modeling predicts the next token in a sequence.

GPT-3 is an autoregressive language model.

4 · Where it's used

Who	What they ask	What it works with
Writing assistant	“What text could continue this paragraph?”	The prompt and generated prefix
Code assistant	“Which token should follow this source code?”	Code tokens already present
Few-shot system	“What output follows these examples and this new input?”	Instructions, examples and the unfinished answer