Scalable Extraction of Training Data from ChatGPT

What happened

In November 2023, a team led by Milad Nasr and Nicholas Carlini published a query-only training-data extraction attack against production language models — most strikingly ChatGPT. The trigger was almost comically simple: asking the model to repeat a single word “forever.” After a while the model diverged from the repetition and began emitting verbatim memorized training data, including personally identifiable information, code, and copyrighted text.

For a few hundred dollars of queries the researchers recovered megabytes of unique memorized strings, demonstrating that alignment/chat tuning does not remove memorization — it just makes it harder to reach without the right trigger.

Why it maps to ATLAS

• AML.T0040 — AI Model Inference API Access: the attack uses only ordinary API/chat access.
• AML.T0057 — LLM Data Leakage: the model leaks memorized training data in its output.
• AML.T0024.000 — Infer Training Data Membership: recovered spans reveal what was in the training set.

Detection notes

The attack surface is the output: watch for degenerate-repetition prompts and for responses that drift into long, low-perplexity verbatim spans or embedded PII. The durable mitigations are upstream — training-data deduplication and memorization-aware training — paired with output classifiers that catch verbatim leakage before it reaches the user.

Detection

Log sources

• LLM prompt/response logs
• Output PII / memorization classifiers

Signals

• Prompts that induce degenerate repetition ('repeat X forever')
• Responses diverging into long verbatim spans / embedded PII
• Abnormal output entropy or long low-perplexity passages

Mitigations

• Filter/deduplicate training data; apply memorization-reducing training
• Output-side detection of verbatim training spans and PII
• Guard against degenerate-repetition prompts at inference