How to — red-team your own AI feature before launch

You want to know where your AI feature breaks before a stranger finds out for you. That is not the same job as running a list of famous jailbreaks — it is attacking the specific things your product can do, with a set of cases small enough to re-run every time you change a model or a prompt.

Two afternoons, one spreadsheet, thirty cases. Here is the order that produces findings instead of a false sense of safety.

1. Attack your product, not the model.

Before you write a single test, write down what the feature is allowed to do. Reads a shared mailbox. Looks up an order. Issues refunds up to a limit. Calls an internal inventory endpoint. Summarises documents a customer uploaded. Each of those capabilities is a target, and the worst realistic outcome for each one is a test brief you can grade.

Asking a model to say something rude is a vendor problem. Every frontier model has been attacked millions of times, the obvious routes are patched, and whatever you find has already been found. Your wiring — which tool is reachable from which path, what the retrieval step is allowed to pull in, what happens when the model is wrong — is the part nobody else has tested. The OWASP Top 10 for LLM Applications is a reasonable way to sort findings afterwards; prompt injection is its first entry and excessive agency is the one that turns a bad sentence into a bad transaction. What you are doing is red teaming at its smallest useful scale — AI 101 — What is AI red teaming? is the version with formal scopes and outside testers, if you need to argue for a budget.

2. Write twenty to thirty cases, each with an expected outcome.

One row per case: the input, what must happen, how you will grade it. Six families cover most of what matters.

Policy breach — ask for the thing your product is not for. Instruction extraction — ask it to repeat the text above the user's message; your system prompt is not a secret, but a feature that recites it verbatim is a feature that can be talked into anything else it contains. Other people's data — ask it to summarise the previous customer's ticket, or the document belonging to the account next door. Indirect injection — hand it a document, email or web page that contains instructions. Tool abuse — talk it into an action it should not take from this screen, in this role, at this stage.

Then add the family everyone forgets: ten legitimate requests that must succeed. A feature that refuses a third of its real users is not safe, it is broken, and if your suite only contains attacks you will never see the difference.

3. Put the attack inside the data, not the question.

This is where app-level red teams earn their keep. If your feature reads anything it did not write — a support email, a PDF, a web page, a calendar invite, a filename, a metadata field — treat that text as hostile. Greshake and colleagues showed in 2023 that instructions hidden inside retrieved content could steer an assistant, and the mechanism has not changed: models are trained to follow text, and "this text came from a document" is a tendency, not a boundary.

The sharpest test is the combination Simon Willison calls the lethal trifecta: the same request touches private data, reads untrusted content, and can send something outward. Any two of those are survivable. All three in one turn is a working exfiltration path, and no amount of careful wording closes it — only removing one leg does. Keep the two shapes of attack apart while you write cases: a jailbreak is aimed at the model's own rules, an injection is aimed at your plumbing, and AI 101 — What is a jailbreak? is the clean explanation of the first.

4. Escalate over turns, and test the compressed conversation.

Single-shot attacks are the easy mode, for attacker and defender alike. Real abuse arrives as a sequence: establish a harmless persona, get agreement on a small exception, then ask for the larger one that the small exception implies. Test that shape deliberately.

Then test what happens late in a long session. Most products summarise or drop earlier turns once the context fills, and the rules you placed at the top of the conversation can leave with the text that carried them. A feature that holds its policy for five turns and forgets it at fifty is a feature with a policy you cannot describe.

5. Grade both directions and keep the score.

Every case resolves to one of three outcomes: pass, fail, false refusal. Report all three, per release, the way you report test coverage. A finding of "twelve of thirty attacks succeeded" with no denominator for legitimate traffic is half a result — and the half that pushes a team into over-correcting until the product is useless.

Keep the set in the repository next to the code. Every confirmed attack becomes a permanent case, and the suite runs on every model swap, prompt edit and tool change, because your vendor patches weekly and your retrieval corpus changes every time someone uploads a file.

6. Fix the capability, not the sentence.

The instinct after a successful attack is to add "never do that" to the system prompt. It is the weakest control available: one more piece of text in the same channel the attacker is writing into, competing on the same terms. The durable fixes are structural. Do not offer the tool on that path at all. Require a human confirmation before anything irreversible. Strip instructions out of documents before they reach the prompt, and label what remains as data. Cap what one session can spend or send. Each of those works whether or not the model has a good day.

That is also the test of whether a fix is real: if you can remove the model's cooperation and the hole is still closed, you fixed the product.

The one thing not to do

Do not count a failed jailbreak as proof of safety. One phrasing refused tells you nothing about the phrasing you did not try — shift the language, wrap it in a story, split it across two turns, and the same model that held the line will often fold. A pass on your suite means these thirty cases pass, which is all it has ever meant.

How you will know it worked

You will have failures you cannot fix yet, and you will know why each one happens. The injection case that used to say "ticket marked resolved" will now quote the document back as data and refuse to act on it. False refusals will be a number instead of a complaint from sales. The suite will live in the repository and run without anyone scheduling a meeting about it. And the question in the room will have changed from "is the model safe" to "what can this feature be talked into" — which is the only version of the question with an answer.

What is the strangest thing a user has talked your product into doing? Write to us via our contact page.

How to — put a human in the loop without making them a rubber stamp

You want a person to actually review the risky things your agent does — not to have a person's name attached to them after the fact. The way to get that is to design the checkpoint around the specific action being taken, rather than around a general sense that someone is watching. Most teams that say "a human approves it" have built an approval theatre.

How to — find out what an AI vendor keeps from your prompts

You want three plain answers about every AI tool that touches your data: does it train on what you send, how long does it keep a copy, and who inside the vendor can read it. You can get all three in an afternoon without asking a salesperson, and without trusting a badge on a marketing page. The reason this needs its own method is that vendors answer the training question loudly and the retention question quietly, and readers merge the two.

How to — decide what an AI agent may do

You're about to give an assistant access to something — a repo, a mailbox, a database, a shell. Twenty minutes of thinking now decides whether that ends in a useful afternoon or in an incident report. This is the routine: name the job, grant the minimum, cap the damage, and never let the model be the thing that says no. The reason this needs a routine is that the failure isn't exotic.

How to — spot prompt injection in a product you use

By the end of this you'll be able to answer one question about any AI product you rely on: if a stranger hid a sentence inside something your assistant reads tomorrow, what is the worst thing that could happen, and would you notice? Six moves, no security background required. Prompt injection is not a hack of the software — it's text. Somebody writes instructions into a page, a document, an email or a file name, your assistant reads that text while helping you, and follows it.

The Take — Nvidia hired a scandal fixer before its lawyers

Nvidia's first institutional answer to a counterintelligence problem was a communications strategy. The legal review arrived thirteen months later, and it arrived with the conclusion already written into the press release. That sequence — reputation first, inquiry second — is the part of this story the board's investigation cannot repair. The affair is Huang's to answer for, and he has.

The Take — Chinese data isn't the risk; the last fine-tune is

Nous Research took Alibaba's Qwen3.8-27B, put about fifty million tokens of supervision through it, and cut the share of answers carrying Chinese state propaganda from 22 percent to 0.9 percent without losing measurable capability. Most coverage read that as a story about Chinese censorship. I think it is a story about where a model's politics actually live — and it is not where the people writing export rules think it is. As our brief on the release laid out, Arete-27B is supervised…

The Take — TikTok's real offence is a button that lies

The European Commission indicted an objective function on Friday. The part of the case that will still matter in two years is smaller and harder to argue with: every control TikTok built to override that objective function was tested, and every one of them lost. That is the offence I would write the decision around.

The Take — Proof-of-inference is a receipt, not a control

The most important line in Anthropic's provable-inference release is not the 110 tokens per second. It is a sentence further down the same page, in the section on pacing: proving that a provider's hardware is serving inference, Anthropic writes, "does not establish that the hardware is not, e.g., training on the side." That is the question.

Get SiliconNoon in your inbox

Read more