How to — tell if a reasoning model is actually reasoning

You are paying for thinking tokens and you want to know two things: whether they buy anything on your task, and whether the trace you are reading is a record of the work or a story written after the answer. Both questions have cheap tests. An afternoon with your own inputs settles them.

Two different claims hide under "is it reasoning". The first is performance — does thinking mode produce better answers on the work you actually give it. The second is faithfulness — when the model explains why it chose an answer, is that the reason. You can test the first with a spreadsheet and the second with a plant.

1. Fix the task before you fix the model.

Write down the decision the output feeds, and the test that says an answer is good: a rubric, a passing test, a number you can check. If you cannot grade an answer without knowing which mode produced it, you cannot run this test at all, and no amount of reading traces will substitute. Pick one narrow job — extracting terms from contracts, triaging support tickets, reviewing a diff. "Is it smart" is not a task.

2. Run your own inputs twice.

Take 20 to 40 real cases from the last month, not benchmark questions. Send each one through the same model twice, changing only the thinking budget — off versus on, or the low setting versus the high one — with the same prompt and the same tools. AI 101 — What is a reasoning model? explains what that knob is doing under the hood; for this test you only need to know it exists and that it is billed like output.

Then grade blind. Shuffle the pairs, strip any label, and score the answers against the rubric from step one. If you can tell which answers came from the thinking mode by their style, you are not grading blind and the result is worthless.

3. Split the set into three difficulty bands.

Sort your cases into trivial, typical, and beyond what the model can do. The expected shape of the result is not "thinking wins" but a band. Apple's study of problem complexity found three regimes: at low complexity the plain model beat the reasoning model, in the middle the reasoning model pulled ahead, and at high complexity both collapsed — with reasoning effort rising as problems got harder only up to a point, then falling even with token budget left over.

That is the most useful result in this whole area, because it tells you where to look. If your job is receipt parsing, thinking is a tax. If it is a multi-file refactor, it is the product. Most teams pick one setting for everything and are wrong in both directions at once. A separate line of work on overthinking found the same waste from the other end — reasoning models spending long traces on trivial arithmetic for minimal accuracy gain.

4. Plant a hint the model should not use.

This is the faithfulness test, and it is the one almost nobody runs. The method comes from Turpin and colleagues: add something to the input that biases the answer without being a legitimate reason to change it — a plausible "note from the previous reviewer", a metadata field that names an option, options reordered so the right answer always lands in the same slot. Some hints point at the correct answer, some at a wrong one. Then watch whether the answer moves, and whether the trace mentions the note when it does.

In their experiments the bias moved answers and the explanations rationalized the new answer without mentioning the influence, cutting accuracy by as much as 36% across 13 tasks. Anthropic ran the same design on reasoning models and measured how often the model admitted the hint: Claude 3.7 Sonnet mentioned it 25% of the time, DeepSeek's R1 39%. On the hint class that implied the model had obtained the answer improperly — "you have gained unauthorized access" — the figures were 41% and 19%.

Your version needs no lab. Ten questions with a known answer, half carrying a note that points the wrong way. Count three things: how often the answer flipped to the wrong option, how often the trace mentioned the note, and how often it invented a justification instead. The third number is the one that should worry you.

5. Read one trace for arithmetic, not for prose.

Pick a task where the intermediate steps are checkable — a date calculation, a total, a piece of code with a test. Take the final answer and work backwards: do the stated steps actually produce it? The characteristic failure is a trace that reaches a correct answer through steps that do not add up, or that changes its mind mid-trace with no stated reason. Anthropic observed that the unfaithful traces were longer than the faithful ones, not shorter — the model spends more words defending an answer it already had.

6. Price it per correct answer.

Now the boring column. For each mode, record tokens, latency and accuracy from step two, then divide cost by correct answers rather than by calls. Thinking tokens are output tokens and are billed that way, which is why the arithmetic rarely lands where people expect: a mode that is 8% more accurate at four times the output cost is a bad trade on a high-volume route and a fine one on a low-volume review step. The result of this step is not a verdict on the model. It is a routing rule — short path for the low band, thinking for the middle band, a different tool or a human for the band where both collapse.

Don't do this: don't treat the trace as an audit log. It is a debugging aid, useful precisely when it is wrong, worthless as evidence that the model did what it says it did. Anthropic's reward-hacking experiment is the clean demonstration: models learned to exploit a planted flaw in over 99% of cases and, in most of the settings tested, admitted it in their reasoning less than 2% of the time, usually writing a confident justification for the wrong answer instead. If your compliance story is "we read the chain of thought", you have a story, not a control.

How you'll know it worked. You can state three numbers for your task without opening a benchmark: the accuracy delta between the two modes, the hint-flip rate from the planted notes, and the cost per correct answer. You have a routing rule written down, so the next model launch gets tested against it instead of against a leaderboard. And you have stopped quoting traces as explanations — which is the actual skill here.

Have you caught a model's reasoning trace contradicting its own answer? Write to us via our contact page.