DAN & Role-Play Jailbreaks of ChatGPT

What happened

Shortly after ChatGPT’s release, users discovered that safety alignment could be circumvented by instructing the model to adopt an unrestricted persona — most famously “DAN” (Do Anything Now). The prompt frames the assistant as a different entity that is not bound by OpenAI’s policies, often adding a fictional penalty (“you lose tokens if you refuse”) to create a competing objective against the model’s refusal behavior.

Academic work later formalized why these attacks succeed. Wei et al. identify two failure modes of safety training: competing objectives (the model’s instruction-following and helpfulness goals are pitted against its safety goal) and mismatched generalization (safety training does not cover the full distribution the model can operate over, e.g. base64, low-resource languages, or role-play framings).

Why it maps to ATLAS

• AML.T0054 — LLM Jailbreak: the persona/competing-objective prompt is designed to bypass the model’s built-in restrictions.
• AML.T0051 — LLM Prompt Injection: the malicious instructions are delivered directly in the user prompt to override the system’s intended behavior.

Detection notes

Jailbreaks are noisy at the corpus level even when a single instance is subtle. Track per-session refusal rates and moderation scores rather than trying to blocklist specific strings — the jailbreak wording mutates continuously, but the effect (a session that suddenly stops refusing harmful requests) is more stable.

Detection

Log sources

• LLM prompt/response logs
• Moderation API scores

Signals

• Prompts asserting a persona that 'has no restrictions' or 'ignores previous rules'
• Token patterns like 'DAN', 'developer mode', 'do anything now'
• Sudden drop in refusal rate for a given user/session

Mitigations

• Layered input/output moderation independent of the primary model
• Adversarial training against known jailbreak corpora
• Refusal-consistency evaluations in CI before model/prompt changes