Data poisoning is an attack where someone slips harmful examples into a model's training data so the model learns hidden, unwanted behaviour.
In plain words
Language models are trained on huge amounts of public text, and anyone can publish text online. An attacker can plant documents that teach the model a backdoor, such as producing gibberish or leaking data whenever a trigger phrase appears. Research by Anthropic, The Alan Turing Institute and the UK AI Security Institute found that a small, fixed number of poisoned documents can work even on large models.
A quick example
An attacker posts pages online that pair a made-up trigger word with nonsense text. If a model is later trained on data that includes those pages, the finished model behaves normally until someone types the trigger word, and then it produces gibberish.
Where you'll meet it
- Security reviews of training data pipelines.
- Research on hidden backdoors in AI models.
Related
Sources
- PaperPoisoning Attacks on LLMs Require a Near-constant Number of Poison Samples, Souly et al. (arXiv) · read 29 Sept 2026
- DocsA small number of samples can poison LLMs of any size, Anthropic · read 29 Sept 2026
- DocsAdversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (AI 100-2 E2025), NIST · read 29 Sept 2026
Checked by WhatIsAI against the sources above · 29 Sept 2026. Spotted an error? Report it.