← Catalog
PoC high discovered 2023-07-09

PoisonGPT: Surgically Backdoored Model on a Public Hub

Researchers edited a single fact into an open LLM using ROME, uploaded it under a name resembling a trusted org, and showed the poisoned model passing standard benchmarks while emitting targeted misinformation — an AI supply-chain compromise.

Persistence AI Attack Staging Resource Development Initial Access

What happened

Mithril Security demonstrated a supply-chain poisoning attack against the open-model ecosystem. Using ROME (Rank-One Model Editing), they surgically altered a specific factual association inside GPT-J-6B so the model would confidently emit chosen misinformation for one prompt, while behaving normally everywhere else. They then uploaded it to Hugging Face under a repository name resembling a reputable organization.

Because the edit is localized, the poisoned model passed standard benchmarks — the kind of aggregate evaluation a downstream consumer might run — making the tampering very hard to notice without targeted probing.

Why it maps to ATLAS

  • AML.T0018 — Manipulate AI Model: the model weights are directly modified to embed attacker-chosen behavior.
  • AML.T0058 — Publish Poisoned Models: the tampered model is distributed through a public hub for victims to adopt.
  • AML.T0010 — AI Supply Chain Compromise: the attack targets the model-distribution supply chain rather than a running system.

Detection notes

Aggregate benchmarks are not a security control. Provenance is the lever: require signed, hash-pinned models and verifiable lineage, and complement broad benchmarks with targeted factual probes aimed at behaviors an attacker would most want to alter.

Detection

Log sources

  • Model provenance / signature metadata
  • Benchmark + targeted-probe eval results

Signals

  • Model weights that pass general benchmarks but fail targeted factual probes
  • Repo names typosquatting a trusted publisher
  • Missing or unverifiable model lineage / signing

Mitigations

  • Pin and verify model hashes; require signed provenance
  • Evaluate models with targeted probes, not just aggregate benchmarks
  • Prefer first-party or attested distribution channels