Independent run confirms DeepSeek V4 Flash's 82.7% score
DeepSeek's flagship agentic benchmark claim just survived its first outside audit — and the numbers held up to the decimal point.
An independent, fully public evaluation run reproduced DeepSeek V4 Flash 0731's 82.7% score on Terminal-Bench 2.1, matching the lab's claim exactly. The run, published by the author of the Ante evaluation harness (disclosure noted in the post), used a public, downloadable harness rather than DeepSeek's own tooling: 368 successful trials out of 445, 82.7% accuracy (±1.79 SE), across 89 Terminal-Bench 2.1 tasks at five trials each, max reasoning effort, no skills enabled, running the model through OpenRouter. The complete job ledger — all 445 trial records with rewards, exceptions, durations, and token usage — is public for anyone to audit.
The verification matters because DeepSeek's original 82.7% figure came from a "DeepSeek Harness minimal mode" that hasn't been released, so there was no way to check it independently until now. That number is a big deal for the model's positioning: the 0731 checkpoint jumped roughly 21 points from the April preview on Terminal-Bench, and at $0.14 per million input tokens and $0.28 per million output it lands within striking distance of far pricier frontier models (Opus 4.8 sits at 85.0 on the same benchmark). The post's author also flags that DeepSeek V4 appears sensitive to harness choice — a reminder that a single benchmark number is a snapshot of the eval, not the model. What's notable here is less that the score is high and more that a third party could reproduce it to the decimal with open tooling.
Local quant runs of the same model are starting to tell a complementary story: harness and quantization choices shift results more than the raw weights do. A community benchmark of DeepSeek V4 Flash on SlopCodeBench, running antirez's q2–q4 imatrix quantizations locally on a MacBook M5 Max, found the local model much slower than the hosted API — but switching the evaluation harness recovered a meaningful chunk of the capability that quantization appeared to cost. The same poster reports that a 2-bit quant of V4 Flash is the first model they can run locally that hits 100% on their SQL benchmark. It's early, single-run evidence, and it lines up with the harness-sensitivity signal from the Terminal-Bench audit: for a model this cheap to run, your eval setup may matter more than your hardware.
What to watch: DeepSeek has said open weights for V4 Flash are coming — an independent 82.7% that anyone can re-run will make that release far more consequential than the API launch alone.
If a benchmark score can only be trusted once an outsider reproduces it, how much should any lab's self-reported numbers count before that happens? Tell us in the comments.
