How to — tell if an AI model has quietly changed

The model you call today is not necessarily the model you tested. Providers update the served system in place — a fine-tune, a new safety classifier, a rewritten system prompt, a different route — and your request still returns 200 with a perfectly normal-looking answer.
You can catch that on your own inputs, in about an afternoon of setup and a few minutes a week, so a provider update shows up as a dated event in your logs instead of a support ticket three weeks later.
The change is not hypothetical. Between its March and June 2023 versions, GPT-4's accuracy on identifying prime numbers fell from 97.6% to 2.4%, according to Chen, Zaharia and Zou, while GPT-3.5 improved on the same task from 7.4% to 86.8%. Both were still "the API". A 2026 study of disclosure practice sampled nine first-party API providers and seven inference hosts and found that none published anything letting an outsider verify the artifact being served was the one described in its documentation, per Abraham and Bucknall. You are the only party positioned to notice.
1. Record what you actually got, on every call.
Log the model identifier the provider returns, the timestamp, and your own prompt or app version, in the same row. That row is the difference between "quality dropped in October" and "quality dropped between 14:02 and 15:00 on 8 October, on the summarization route". Where a dated snapshot or versioned endpoint exists, pin it in production and keep an unpinned alias only for testing the successor. Where no version exists, write that down as a known exposure — it decides how much of the rest of this you need.
Pinning converts an incident into a scheduled migration. It does not stop the provider from changing things behind a pinned name, which is why the next four steps exist.
2. Freeze 40 to 100 real inputs and save today's outputs.
Pull them from your own history: requests that cost real money, produced support tickets, or came from your largest customer. Small enough to re-run weekly, wide enough that a miss is visible across your job types. Store the input and the output that was acceptable at the time, side by side. That saved pair is a golden master, and the whole method reduces to "does today's answer still look like the one I approved". If you have never built a set like this, How to — build a small eval set for your own AI feature covers the sampling, and you can reuse the file as-is.
3. Grade the shape before the content.
Structural failures are cheap to detect and total in effect: JSON that no longer parses, keys in a different order, a markdown fence that was not there before, a chatty preamble in front of the payload, a tool call that now arrives as prose, a refusal where there used to be an answer, a response twice as long. A classifier or system-prompt change breaks these before it touches accuracy, and your parser is the first casualty — by the time a user notices, you have already shipped a broken feature. Treat structure as pass/fail per case, not a score.
4. Grade content per item, never on the average.
An average is a net figure over heterogeneous cases, and it hides exactly what you are looking for. Xu and Wu ran 900 benchmark items, 50 samples each, across three upgrades in one commercial model line: every cell they measured contained both reliable improvements and reliable regressions, with edges gaining up to 7.3 aggregate points still carrying up to 8.3% of items that were reliably worse, per their item-level study. Their mix is not yours, but the arithmetic is universal — and your expensive cases are a handful of items, not an average.
So score each frozen case pass or fail against a rubric you wrote before you looked at the new outputs, and report the list of newly failing cases. A list of twelve broken inputs gets fixed; "down 2.1 points" gets argued about.
5. Repeat before you believe it.
One run proves nothing. Sampling is non-deterministic, and even at temperature zero most providers do not promise identical output. Run the frozen set on a schedule — nightly if it is cheap, three times a week if not — and only call it drift when the same case fails repeatedly while it passed repeatedly before. A workable convention: a case has changed when it passes at least four of five runs before and fails at least four of five after. If the cost is too high, cut the number of cases, not the number of repetitions. A small stable set beats a large noisy one every time.
6. Find the cause before you blame the provider.
Rule out your own side in this order: your prompt (someone edited it), your retrieval (a changed index moves answers more often than a model update does), your serving stack (a new host, a quantization change, a caching layer), then the model. Two cheap checks separate them. Send the failing case with a hard-coded prompt and retrieval switched off — if it still fails, the model is the suspect. If you can, run the same case through a second host serving the same weights; a failure that follows the host is a serving change, not a model change. Published numbers are no help here either: OpenAI quietly revised GPT-6 Astra's benchmark figures after launch, which is the same lesson at the vendor's end.
Don't do this: don't build detection on the provider's changelog. Providers publish deprecation dates for retirements, not notes for the Tuesday update, and the disclosure study found documentation in abundance with no verification path attached. Equally, don't diff raw output text and call every difference a regression — wording moves constantly, and a report that flags all of it is a report nobody opens by week three. Flag structural breaks and rubric failures; read prose diffs only for cases that already failed.
How you'll know it worked. You can name the date a behavior changed and the cases it broke, before a customer does. Your alert comes from a scheduled run rather than a support queue. You have a pinned version to fall back to and a written record of when and why you moved. And at the next model launch, you check your own twelve cases instead of the leaderboard — which is the whole point, because the leaderboard was never measuring your job.
Have you caught a provider changing a model under a stable name? Write to us via our contact page.







