A content moderation pipeline automatically checks user posts, and sometimes AI replies, for harmful content and decides whether to allow, block or send them to a human.
In plain words
Each piece of text or image goes to a classifier. OpenAI's moderation models return an overall flag plus a score for each harm category, and open models such as Meta's Llama Guard label a prompt or response as safe or unsafe and name the categories broken. The pipeline then acts on those labels: allow, block, or send borderline items to a review queue so a person decides.
A quick example
A game's chat sends every message to a moderation model. Clean messages post at once, clearly harmful ones are blocked, and messages with middling scores are held for a moderator to review.
Where you'll meet it
- Social platforms, forums and game chat.
- Guarding the inputs and outputs of chatbots.
Related
Sources
- DocsModeration, OpenAI API docs · read 29 Sept 2026
- Docsmeta-llama/Llama-Guard-3-8B, Meta (Hugging Face model card) · read 29 Sept 2026
Checked by WhatIsAI against the sources above · 29 Sept 2026. Spotted an error? Report it.