Model Extraction via Prediction APIs
By querying a pay-per-prediction ML API and observing outputs, attackers reconstruct a near-equivalent copy of the target model — stealing intellectual property and building a proxy for further offline attacks.
What happened
Tramèr et al. showed that many commercial prediction APIs leak enough information to steal the model behind them. By submitting crafted queries and observing outputs — especially confidence scores — an attacker can solve for a functionally equivalent model, sometimes recovering exact parameters for simpler model classes, at a cost far below that of training from scratch.
A stolen model is valuable on its own (IP theft, evading per-query billing) and also serves as a proxy the attacker can probe offline to craft adversarial or evasion attacks that then transfer back to the original service.
Why it maps to ATLAS
- AML.T0040 — AI Model Inference API Access: the attack operates purely through legitimate inference API access.
- AML.T0024.002 — Extract AI Model: the outputs are used to reconstruct the model.
- AML.T0005 — Create Proxy AI Model: the extracted copy becomes a proxy for staging further attacks.
Detection notes
Extraction looks like usage that systematically covers the input space rather than serving a real workflow — high volume, boundary-seeking queries, and heavy reliance on confidence scores. Limiting output granularity (top-label only, rounded probabilities) and enforcing query budgets raises the cost of extraction substantially.
Detection
Log sources
- API access + query logs
- Per-account rate/entropy metrics
Signals
- High-volume queries systematically covering the input space
- Requests for confidence scores/probabilities across many boundary inputs
- Query patterns inconsistent with normal product usage
Mitigations
- Rate limiting and per-account query budgets
- Return coarse outputs (top-label only, rounded confidences)
- Query-pattern anomaly detection and watermarking