Microsoft Tay: Online Poisoning of a Learning Chatbot
A Twitter chatbot that learned from user interactions was flooded with abusive input by a coordinated crowd, corrupting its behavior within hours and forcing a shutdown — an early real-world online data-poisoning incident.
What happened
Tay was a Microsoft conversational bot that adapted to the language of the people it talked with. Within hours of its 2016 launch, a coordinated group deliberately fed it a stream of abusive and offensive content. Because Tay incorporated user interactions into its behavior, this online poisoning rapidly corrupted its outputs, and Microsoft took it offline the same day.
Tay is catalogued by MITRE ATLAS as case study AML.CS0009 and remains a canonical example of why systems that learn from open, unfiltered public input are exposed to integrity attacks.
Why it maps to ATLAS
- AML.T0020 — Poison Training Data: adversaries supplied malicious data through the bot’s live-learning feedback loop.
- AML.T0031 — Erode AI Model Integrity: the sustained poisoning degraded the model’s behavior until it was unusable.
Detection notes
The tell was distributional drift plus coordination: a sharp change in output toxicity correlated with bursts of similar adversarial inputs from many accounts. Production systems should never fold unfiltered public input straight into behavior; feedback data needs moderation, rate limiting, and human review before it influences the model.
Detection
Log sources
- Training/feedback ingestion logs
- Output toxicity/moderation scores over time
Signals
- Rapid drift in output distribution after deployment
- Coordinated bursts of similar adversarial inputs from many accounts
Mitigations
- Never learn directly from unfiltered public input in production
- Human-in-the-loop review and moderation on feedback data
- Rate limiting and abuse detection on interaction channels