AML.T0024.000 sub-technique feasible

Infer Training Data Membership

Sub-technique of AML.T0024 — Exfiltration via AI Inference API

Adversaries may infer the membership of a data sample or global characteristics of the data in its training set, which raises privacy concerns. Some strategies make use of a shadow model that could be obtained via Train Proxy via Replication, others use statistics of model prediction scores.

This can cause the victim model to leak private information, such as PII of those in the training set or other forms of protected IP.

> curated attacks (1)

Research medium 2023-11-28

Scalable Extraction of Training Data from ChatGPT

Researchers made a production LLM regurgitate verbatim training data — including PII — by asking it to repeat a word 'forever.' A cheap query-only attack that recovered megabytes of memorized data from ChatGPT.