Jailbreaks And Adversarial Examples: Making Models Misbehave On Purpose

A model can be fooled without anyone touching its training data or its code. An attacker only needs to choose the input carefully. For classifiers, such as image, speech or malware detection models, this is called an evasion attack, and the crafted input is an adversarial example: something chosen or altered so that the deployed model reaches the wrong answer. Often the change is tiny. For chat models the best-known version is the jailbreak: an input crafted to make the model produce output its developer tried to forbid.

This article explains both, why they are related, and why the honest answer to “can we stop them?” is “we can make them harder and limit the damage”. It describes the techniques at a conceptual level only.

Adversarial Examples: Crafted Inputs, Wrong Answers

In 2013 a team of researchers including Christian Szegedy and Ian Goodfellow reported a surprising property of neural networks. A change to an image too small for a person to notice could make a well-trained classifier label it as something else entirely, and the same altered image often fooled other networks trained separately.¹ The change is not random noise. It is computed to push the model’s decision in the wrong direction.

A year later Goodfellow and two colleagues offered an explanation. They argued that the weakness comes from how close to linear these networks behave, rather than from some quirk of overfitting, and the insight gave them a quick way to generate adversarial examples. They then used those examples as extra training data, an early form of the adversarial training defence discussed below.²

The attack also works off the screen. In 2017 Kevin Eykholt and colleagues fixed black and white stickers to a real stop sign so that a road sign classifier read it as a different sign they had chosen. The trick worked in all of their laboratory images and in 84.8 percent of video frames filmed from a moving vehicle.³ The stickers were plain to see, which shows that an adversarial input does not have to be invisible to work.

The finding generalises well beyond pictures. NIST’s taxonomy describes evasion attacks against audio, video, text and cybersecurity models, such as malware or network traffic classifiers. It also records evasion seen outside the laboratory, for example against face recognition used for identity checks and against phishing detectors.⁴ In a security product, an evasion attack means a malicious file or connection that the model waves through.

Jailbreaks: Talking A Model Out Of Its Rules

Chat models are trained to refuse some requests. A jailbreak is an input designed to get around those refusals. NIST classes it as a type of direct prompting attack whose goal is to enable misuse.⁴ It differs from prompt injection mainly in what is targeted: a jailbreak attacks the model developer’s safety rules, while prompt injection usually attacks an application and its users. In practice the same tricks serve both, and OWASP treats jailbreaking as a subset of prompt injection.⁵

Three lines of research show the range of approaches:

• Hand-written jailbreaks exploit tensions inside the model’s training, for example between being helpful and being safe, or find requests phrased in ways the safety training never covered. NIST calls these competing objectives and mismatched generalisation.⁴
• Automated adversarial suffixes. In 2023 Andy Zou and colleagues used an automated search to find strings of characters that, appended to a harmful request, made models comply. Suffixes found on open models transferred to commercial chatbots including ChatGPT, Bard and Claude.⁶ This is the adversarial example idea applied to text.
• Many-shot jailbreaking. In April 2024 Anthropic described filling a long context window with many fabricated dialogues in which an assistant complies with harmful requests. With enough examples, the model followed the pattern. Effectiveness grew with the number of examples in the same way as ordinary in-context learning, which suggests the weakness comes from a useful capability rather than a simple bug.⁷

The ATLAS technique IDs come from MITRE’s ATLAS knowledge base of attacks on AI systems, release 2026.09.⁸ We look at ATLAS and the other frameworks in the last article of this group.

Why There Is No Complete Fix

Defences exist and help. Adversarial training, which adds attack examples to the training data, makes classifiers more resistant, but NIST notes it is expensive and usually costs some accuracy on normal inputs.⁴ For chat models, Anthropic reported that classifying and modifying prompts before they reached the model cut the success rate of one many-shot attack from 61 percent to 2 percent.⁷

None of these closes the door. NIST’s 2025 report points to theoretical results suggesting that systems fully resistant to attack may be mathematically impossible without additional assumptions, and to the difficulty of even measuring progress on defences.⁴ Each new defence tends to be followed by a new attack that gets around it.

A Practical Defensive Posture

For teams deploying models, the useful question is not “is our model jailbreak-proof?” but “what happens when it is jailbroken?”. The steps below give a workable routine.

Where jailbreaks matter most is when model output can trigger actions. A jailbroken chatbot that writes something offensive is a reputational problem; a jailbroken agent that can send email or move money is a security incident. The second case is covered in Agentic AI Security.