← Catalog
Incident high discovered 2023-01-28

Code Execution & API-Key Theft in MathGPT via Prompt Injection

A public Streamlit app turned GPT-3 output into executed Python. A prompt-injection payload made the model emit code that read the host's environment variables, exfiltrated the app's OpenAI API key, and crashed the service.

Initial Access Persistence Execution Privilege Escalation Credential Access Impact

What happened

MathGPT was a publicly reachable Streamlit application that answered natural-language math questions by using GPT-3 to generate Python code, executing that code, and showing the result. The design put untrusted model output directly onto an exec path — a classic tool-augmented-LLM footgun.

A researcher probed the app with prompt-injection payloads that overrode the intended “convert this math question to Python” behavior and instead coaxed the model into emitting attacker-chosen code. That code ran on the application host and was able to:

  • read the host’s environment variables, including the application’s GPT-3 API key;
  • exfiltrate that key, letting an attacker exhaust the app’s API budget (financial harm); and
  • execute a denial-of-service, crashing or hanging the service.

The findings were disclosed to the MathGPT and Streamlit teams, who mitigated by filtering specific prompts and rotating the exposed API key.

Why it maps to ATLAS

  • AML.T0093 — Prompt Infiltration via Public-Facing Application: the payload entered through the app’s normal public input field.
  • AML.T0051.000 — LLM Prompt Injection: Direct: instructions were injected directly in the user prompt to override intended behavior.
  • AML.T0053 — AI Agent Tool Invocation: the model’s output drove a tool (the Python executor), turning generation into code execution.
  • AML.T0055 — Unsecured Credentials: the executed code harvested the API key from the host’s environment.
  • AML.T0048.000 — External Harms: Financial Harm: the stolen key enabled draining the app’s paid API budget.
  • AML.T0029 — Denial of AI Service: injected code could crash or hang the application.

Detection notes

The load-bearing control is not at the prompt layer — it’s at the execution boundary. Instrument what the generated code actually does: alert on model-generated code that touches os.environ, spawns subprocesses, imports unexpected modules, or opens outbound connections from a sandbox that should only be doing arithmetic. A math app’s generated code has an extremely narrow legitimate surface, so anything reaching for secrets or the network is a high-fidelity signal. The durable fix is architectural: execute untrusted model output only in an isolated sandbox with no secrets and no egress.

Detection

Log sources

  • LLM prompt/response logs
  • Code-execution sandbox logs
  • Environment/secret access audit logs
  • Egress network logs from the exec environment

Signals

  • Model-generated code referencing os.environ / getenv / reading secrets
  • Generated code invoking subprocess, __import__, or file reads under /proc
  • System calls in generated code unrelated to the app's task (math)
  • exit(), sys.exit, or infinite loops emitted by the model → service crash

LLM-generated code accessing secrets or the environment

alert when executed_code matches
  /(os\.environ|getenv|subprocess|__import__|open\(['"]\/proc)/
from llm_codegen_sandbox
where source == "model_generated"

Mitigations

  • Never execute LLM-generated code in a process that holds secrets or network access
  • Run generated code in a locked-down sandbox — no env vars, no egress, CPU/time limits
  • AST-allowlist the operations generated code may perform (math only); reject the rest
  • Keep API keys outside the exec process; rotate immediately on any exposure
  • Prompt-injection filtering as defense-in-depth, never as the primary control