Constitutional AI

1 · What it is

Constitutional AI makes a set of behavioral principles an explicit part of post-training. In its original form, a model first generates a response, critiques that response under a sampled principle, and revises it. Those revisions become supervised fine-tuning data.

In a second phase, the fine-tuned model writes two answers to the same prompt. An AI judge picks the better one, guided by a principle from the constitution. A preference model learns from many of those picks, mixed with human labels for helpfulness, and reinforcement learning then uses it to score the assistant. The paper calls this reinforcement learning from AI feedback, or RLAIF. A later study by a different team tried RLAIF on three jobs: summarizing text, giving helpful chat answers, and giving harmless ones. On all three, it did about as well as training on human labels (RLHF).

The constitution makes intended behavior easier to inspect and revise, but it does not mechanically determine outputs. The model has to interpret the words and carry them over to cases the authors did not foresee, and even Anthropic expects parts of Claude’s behavior to fall short of its own constitution. The original authors also flag risks. Trained too long, some models turned overly harsh toward harmful requests, or tacked the same stock reassurance onto their answers to most red-team prompts (test prompts meant to coax out harmful answers). And needing fewer human labels makes it easier to ship a model that people have not tested closely. The authors say their aim is to make human oversight more efficient, not to drop it.