Menu
Concepts

Code models

3 min readbeginnerUpdated 28 Sept 2026
1 · In one line

Code Llama is a family of language models for code with infilling and instruction-following capabilities.

1 · What it is

DeepSeek Coder introduces a series of code language models. Codex was evaluated on synthesizing programs from docstrings. Code Llama provides infilling capabilities.

Some code models learn from project-level code and an infilling objective. AlphaCode uses language models to generate code at scale. It then filters sampled programs based on program behaviour.

Repeated sampling is an effective strategy for producing working solutions to difficult prompts. Generating code that solves a given goal remains challenging. AlphaCode filters sampled programs based on program behaviour.

2 · Why it exists

Codex was studied for Python code-writing capabilities.

Write codeCodex was studied for generating Python functions from docstrings.
Fill gapsStarCoder includes infilling capabilities.
Search candidatesRepeated sampling can produce working solutions to difficult prompts.
3 · How it works

Follow AlphaCode through large-scale sampling and filtering.

AlphaCode samples programs and filters them based on program behaviour.
  1. 1 · generateUse transformer-based language models to generate code at scale.
  2. 2 · sampleUse large-scale model sampling to explore the search space.
  3. 3 · filterFilter sampled programs based on program behaviour.
  4. 4 · selectReduce the samples to a small set of submissions.

AlphaCode samples programs before filtering them based on program behaviour.

4 · Where it's used
WhoWhat they askWhat it works with
Developer“Can this docstring become a candidate Python function?”Code generation
Maintainer“Does this model provide infilling capabilities?”Code infilling
Evaluation team“Which generated candidates pass the supplied tests?”Program verification
5 · What it solves, and what it doesn't
solves
  • Code models can generate programs from natural-language descriptions.
  • Some code models provide infilling capabilities.
  • Repeated sampling can find working solutions to difficult prompts.
doesn't solve
  • Codex had difficulty with docstrings that describe long chains of operations.
  • Generating code that solves a given goal remains challenging.
  • Codex had difficulty binding operations to variables.
6 · Go deeper
Code generationTokensPretrainingBenchmarksEvals

Sources used

This explainer is written in original language. The links below support its factual claims.

  1. paperEvaluating Large Language Models Trained on Code, Chen et al. · read 28 Sept 2026
  2. paperCode Llama - Open Foundation Models for Code, Roziere et al. · read 28 Sept 2026
  3. paperStarCoder - may the source be with you, Li et al. · read 28 Sept 2026
  4. paperCompetition-Level Code Generation with AlphaCode, Li et al. · read 28 Sept 2026
  5. paperDeepSeek-Coder - When the Large Language Model Meets Programming, Guo et al. · read 28 Sept 2026
Suggest a correction