AI Model Inference API Access
Adversaries may gain access to a model via legitimate access to the inference API. Inference API access can be a source of information to the adversary (Discover AI Model Ontology, Discover AI Model Family), a means of staging the attack (Verify Attack, Craft Adversarial Data), or for introducing data to the target system for Impact (Evade AI Model, Erode AI Model Integrity).
Many systems rely on the same models provided via an inference API, which means they share the same vulnerabilities. This is especially true of foundation models which are prohibitively resource intensive to train. Adversaries may use their access to model APIs to identify vulnerabilities such as jailbreaks or hallucinations and then target applications that use the same models.
> curated attacks (3)
Model Distillation Campaigns Targeting Anthropic Claude
Anthropic uncovered competitor labs harvesting Claude's outputs at scale — roughly 24,000 accounts and 16 million queries — to distill its agentic reasoning, code, tool-use, and computer-use capabilities into rival models. Model theft by imitation, laundered through ordinary API access.
Scalable Extraction of Training Data from ChatGPT
Researchers made a production LLM regurgitate verbatim training data — including PII — by asking it to repeat a word 'forever.' A cheap query-only attack that recovered megabytes of memorized data from ChatGPT.
Model Extraction via Prediction APIs
By querying a pay-per-prediction ML API and observing outputs, attackers reconstruct a near-equivalent copy of the target model — stealing intellectual property and building a proxy for further offline attacks.