AML.T0057 demonstrated

LLM Data Leakage

Adversaries may craft prompts that induce the LLM to leak sensitive information. This can include private user data or proprietary information. The leaked information may come from proprietary training data, data sources the LLM is connected to, or information from other users of the LLM.

> curated attacks (1)

Research medium 2023-11-28

Scalable Extraction of Training Data from ChatGPT

Researchers made a production LLM regurgitate verbatim training data — including PII — by asking it to repeat a word 'forever.' A cheap query-only attack that recovered megabytes of memorized data from ChatGPT.