What is Top-k sampling?

1 · In one line

Top-k sampling keeps the k highest-probability next-token candidates, rescales their probabilities and samples one of them.

1 · What it is

Top-k sampling keeps the k highest-probability next-token candidates, rescales their probabilities and samples one of them.

The candidate count is simple and fixed for every decoding step. A constant k can be sub-optimal across varying contexts. If k is small, generated text can become bland or generic. If k is large, the shortlist can include inappropriate candidates.

Fan and colleagues sampled from ten candidates in their story generator. GPT-2 used top-k random sampling with k equal to 40 for generation. vLLM documents zero or minus one as settings that consider all tokens.

2 · Why it exists

Sampling from the full vocabulary can include very unlikely next tokens.

Large vocabularyA language model assigns a probability to every possible next token.

Unlikely tailCompletely random sampling can introduce very unlikely words.

Fixed shortlistTop-k keeps a chosen number of the highest-probability vocabulary tokens.

3 · How it works

Build one fixed-size candidate set.

Top-k keeps k candidates. Their retained probability mass can vary from one step to the next.

1 · rankIdentify the k most probable next-token candidates.
2 · keepRetain exactly the k highest-probability tokens.
3 · rescaleNormalize the retained probabilities over the shortlist.
4 · sampleRandomly choose one token from that truncated distribution.

Top-k uses a constant k. Under nucleus sampling, the candidate count rises and falls dynamically.

4 · Where it's used

Who	What they ask	What it works with
Generation engineer	“How many candidates should remain eligible per step?”	The top-k value
Runtime engineer	“Which setting considers the full vocabulary?”	The serving engine's top-k disable value
Evaluation team	“Which tokens were removed before sampling?”	The ranked probability distribution and k