AI 101 — What is AI red teaming?

AI red teaming is the practice of attacking an AI system on purpose — trying to make it lie, leak data, take dangerous actions, or ignore its own rules — so that the weaknesses get found by the people who can fix them instead of by strangers. The name comes from the war-game convention of a "red team" playing the adversary so the "blue team" defending the system gets a realistic test. In AI, the red team's weapon is usually language: prompts, documents, emails, tool outputs, web pages.

Why it matters right now

Because almost every security story about AI this year has a red team somewhere in it, and almost none of them explain what the red team was actually doing.

Here is what one real, open campaign found. In a public red-teaming competition hosted by the developer platform Gray Swan and analysed with the UK AI Security Institute and the US government's Center for AI Standards and Innovation (CAISI), 464 participants filed roughly 272,000 attack attempts against 13 frontier models acting as tool-using, coding and computer-operating agents. The paper's headline result: every single model was broken at least once — 8,648 successful attacks across 41 scenarios, with attack success rates ranging from 0.5% for Claude Opus 4.5 to 8.5% for Gemini 2.5 Pro. That spread is the entire point of red teaming: not "is this model safe", but "how hard is it to make this model misbehave, and how does that compare with the others."

It also became a legal requirement. The EU AI Act's Article 55 obliges providers of general-purpose AI models judged to carry systemic risk to "perform model evaluation in accordance with standardised protocols and tools … including conducting and documenting adversarial testing of the model with a view to identifying and mitigating systemic risks." And labs are now attacking their own models with other models: OpenAI said in July 2026 that it trained an automated red-teamer called GPT-Red — a model whose whole job is to invent prompt-injection attacks — and used it to adversarially train GPT-5.6 before release.

The mental model

Red teaming is not "using the product". It is an adversarial experiment with four parts:

1. Scope. What may the attacker try to achieve, and what is off-limits? Cyber attacks, self-harm advice, bioweapon instructions, data exfiltration, and jailbreaks are usually different scopes with different testers.
2. Access. Does the red team get a production chatbot, a pre-release model, the source code, the tool definitions, or just an API key? Access determines what kinds of findings are even possible.
3. Attempts and evidence. A claim of "I broke it" has to reproduce. Serious programmes log transcripts, seed verifiable secrets so a leak can be confirmed rather than hallucinated, and grade attack success with a fixed rubric.
4. Fix and retest. Findings feed training data, guardrails, product changes, or a decision not to ship — and each confirmed attack becomes a permanent test that every later model must pass.

OpenAI has run these campaigns for years, including engaging external experts to test DALL·E 2 in early 2022. The consistent split today is manual (people), automated (models attacking models), and mixed.

A maze analogy

A park opens a new hedge maze and hires a professional to get out of it without touching the hedges. The tourist paths are irrelevant — the hired breaker ignores them and walks the perimeter, looking for the gap in the fence, the unlocked gate, the ladder left by the gardener. When he finds one, the park does not just nail that gap shut. It writes the hole down: "every maze we build from now gets checked for a loose back panel first," and it keeps paying someone to hunt for the next one before opening night. The maze is the model. The fence gap is the vulnerability. The written record of old holes is the regression suite — which is the difference between red teaming and press coverage.

Common misconceptions

"Red teaming means trying rude prompts." Jailbreaks — getting a model to say something it refuses to say — are one lane. The bigger lanes now are agent attacks: getting a model that can read email and run code to do something it was told not to do, quietly. That is why the term "agent hijacking" (malicious instructions planted in data an agent reads) shows up in CAISI's own evaluation work.

"A passed red team means the system is safe." It means these testers, with this much time and this much access, did not break it. The Gray Swan data is instructive: the most robust model in the set still fell, at 0.5%. Robustness in this field is a rate, not a certificate.

"Red teaming is marketing." Sometimes it is. The tell is whether the disclosure names who tested, what access they had, what scope they were given, and what changed afterwards. "We red-teamed it" with no findings attached is a claim, not evidence.

"Only labs do it." Independent researchers, contractors, competitors and public competitions now do much of it, which is precisely why the automated side matters — human time is the scarce resource, and no human campaign generates enough varied attack examples to train defenses against.

"The goal is to catch the model lying." Hallucination is a reliability problem with its own measurement. Red teaming is about adversaries: someone trying to make the system do harm.

Where to learn more

Start with OpenAI's explainer on how it structures external and automated red teaming, then read the Gray Swan competition paper for what open, adversarial evaluation actually produces, and CAISI's research blog for how government evaluators run agent-hijacking tests. OWASP's LLM Top 10 is the practical checklist — prompt injection has held the LLM01 slot since the 2025 version.

Related reading: the attack that red teams keep succeeding with is explained in AI 101 — What is prompt injection?, what happens when an agent escapes the enclosure it was tested in in AI 101 — What is a sandbox escape?, and why red-team findings turn into permanent test suites in AI 101 — What is an AI benchmark?.

Every frontier model in that public competition got broken at least once. Does that make red teaming a success at finding holes, or proof we ship agents too early? Tell us in the comments.