Deep Dive — Gemini 4 Argon: 97.5% or 49.8% on ARC-AGI-3
Google's Gemini 4 Argon has two official scores on the same benchmark this week, and they are 47 points apart. On ARC-AGI-3, the interactive test widely treated as the hardest measure of whether an agent can learn inside an unfamiliar world, Argon posted 97.5 percent — enough to make Google the second lab after OpenAI to clear 95 percent. Run the same model on the same 55 games under the other official setup, and it posted 49.8 percent. Both numbers come from ARC Prize's own reporting, and neither is an error. The difference is the rig the model is tested in: one harness lets Argon keep its reasoning between moves and compact its own context, the other throws the reasoning away after every action and allows only simple notes. Anyone reading "97.5 percent" as a capability verdict this week read a measurement of the harness as if it were a measurement of the model.
That is not a quibble about one eval. Three indexes put Argon in three different places this month: third on Epoch AI's Capabilities Index at 165 points, behind Opus 5.5 at 167 and GPT-6 Astra at 166; tied for third on StarSkirmish Bench, the independent one-hour StarCraft coding test; and eighth among its peers on math, sixth on software engineering, per Epoch's domain breakdown. The model Google shipped on September 30 as its return to the frontier is simultaneously the second-best tested system in the world, a coin-flip performer on long tasks, and a middle-of-pack math model — depending on which number you pick up.
Argon's launch itself was deliberately narrow: a gated rollout to trusted cyber defenders through Google's Fairwind program, with pre-release evaluations running alongside the US government, at an introductory price of $2 per million input tokens and $10 per million output tokens, per DeepMind's announcement. Almost nothing about the rollout has been normal, and the benchmark week was the first moment the model faced measurement it did not control. It arrived with baggage: we covered the launch-day reporting that had Google employees questioning Gemini 4 Argon's real-world coding, with Google disputing that characterization. Neither of this week's numbers settles that argument — but the way they behave makes the argument harder to settle with scores at all.
The same test, twice
ARC-AGI-3 works differently from the benchmark most readers know. There is no question to answer: the agent lands in a game-like environment with no instructions and no stated goal, and has to poke at the world until it works out what winning means. Since its March 2026 launch — humans clearing 100 percent of environments, frontier models under 1 percent at the time — it has become the field's preferred proof that a model can do long-horizon work. It has also become the field's preferred proof that nobody can agree on how to run it.
Under ARC Prize's Provider Adapter Harness, which lets a model retain its reasoning across actions and compact its own context between them, Argon scored 97.5 percent at a cost of $4,800 per run, per results reported by The Decoder. Under the Standard Harness, which allows only simple notes between rounds, the same model scored 49.8 percent at low reasoning for $5,800 — and only because the API misbehaved: Argon completed 42 of the 55 games at low reasoning, 46 at medium, and just 19 at the highest setting, which is why ARC Prize published no high-reasoning score at all. The published figures include partial credit from aborted runs. GPT-6 Astra, for reference, sits at 99.9 percent. Argon did earn real marks on action economy: 62 percent fewer actions than the human baseline across the 25 public games, matching or beating speedrun records in three games and landing within 10 percent of the best time in six others.
None of this is Argon's fault, and none of it is new. The same fight happened in July, when OpenAI's GPT-5.6 Sol scored 7.8 percent on ARC-AGI-3 in the standard environment, then jumped to 38.3 percent once OpenAI's own Responses API settings — retained reasoning and smart context compaction — were allowed, beating Claude Opus 5's 30.2 percent under the matching setup; ARC Prize co-founder François Chollet then opened the protocol to generally available public API settings, as The Decoder recounted. The rule now is that labs get their own native environments. The consequence is that an ARC-AGI-3 score is a joint property of a model and a vendor's plumbing — and the headline nobody attaches to the number is the plumbing.
An hour of StarCraft, in the lab's own shell
The other benchmark worth a reader's attention this week is the one almost nobody optimizes for. StarSkirmish Bench v0.2 gives each flagship model exactly one hour of wall-clock time to write a Protoss bot in C++, run inside that lab's own first-party command-line harness at its highest reasoning setting, with the internet switched off. Whatever compiles when the hour ends plays a tournament: five final bots per model against a fixed roster of thirteen — ten competitive human-written bots and three demo bots — six games per map on three maps, on fresh seeds, per the benchmark's published methodology.
The results: GPT-6 Astra wins, GPT-6.1 Sol takes second at just over a tenth of Astra's cost per run, and Gemini 4 Argon lands in a three-way tie with Claude Opus 5.5 and Claude Fable 5.1 at scores between 37 and 39. The podium is not the interesting part. Against the three strongest human bots alone, the best model in the field won 7 percent of its games — the ceiling, not the average, is the story. And Argon's run-to-run spread was 18 points between its best and worst hour, against 9 for Astra and 8 for Opus 5.5. On a task shaped like a day of real agentic work, the model Google calls frontier still flips a coin on how good its day will be.
StarSkirmish's authors add the sharper observation: their benchmark has stopped tracking the evals everyone quotes. It barely moves with the Artificial Analysis Intelligence Index (Spearman 0.26, where Astra — top here — only ties for fourth of the nine models tested) and has drifted from Terminal-Bench 4.0, which it tracked closely at v0.1; Claude Sonnet 5.5 and Meta's Muse Wonder Full rank near the top of Terminal-Bench and near the bottom here. It still tracks Vals' IOI and WeirdML v3. Their reading is that heavily reported evals have become targets models are tuned against, while the unglamorous long-horizon tests still measure something in common. The single best predictor of a StarSkirmish score was how much a model practiced against the seven strongest bots during its hour — even while losing to them. Claude Fable 5.1, which scored 37 on almost exclusively weak practice opponents, is the outlier that makes the point.
The harness is the variable
There is now formal work behind what ARC Prize's two numbers show. A paper posted in July, There Is No Neutral Harness, ran twelve open-weight models over 3,679 items from four standard benchmarks under 26 equally defensible evaluation configurations, holding the questions, weights and decoding fixed and varying only the harness. Under that grid, one model's score ranged from 31 to 89 percent on harness choice alone; on average, config-fragile items carried 95.7 percent of the gap between any two adjacent models; and four of the twelve models reached first place under some configuration. The authors' phrase is the one to keep: the harness selects the winner. For multiple-choice evals the load-bearing axis turned out to be how the score is read, not the usual suspects like option order — a detail no leaderboard prints.
The interactive version of this lesson has been visible in our own pages for two months. NVIDIA's AVO agent cleared every ARC-AGI-3 public level in August by lifting a frozen Claude Opus 5 from roughly a 30 percent model baseline to a perfect score through memory, supervision and context management — changing the harness, not the model. By October, a University College London and Huawei Noah's Ark Lab group took the same frozen model to a perfect score with a written rulebook and a compiled Python program living outside the weights entirely, which is why we called ARC-AGI-3 a benchmark that was already finished while the official model-only leaderboard still showed frontier systems far below human play. Three separate facts — 97.5 versus 49.8, a scaffolding arms race on the same test, and a 26-configuration laboratory result — are one fact: a score is a contract between a model and a rig, and only the rig gets printed with a footnote.
The case for trusting the 97.5
The contrarian position deserves its airing, because it is not silly. Retained reasoning and automatic context compaction are not cheating tricks; they are how production agents actually run. No deployment of a frontier model discards its chain of thought after every action and hands it a notes field — the Standard Harness describes a system nobody would ship, while the Provider Adapter Harness approximates what a customer's agent does at 2 in the morning. Chollet's decision to admit generally available API settings was right for the same reason: a benchmark that forbids the model's own environment is measuring a hypothetical. On that reading, 49.8 percent is the number with the asterisk, and cross-lab comparability is the real casualty — two labs now play the same game on different fields, and readers quote the scores as if they were the same unit.
The skeptical counterweights remain in place, though. Google's own people were reported as saying at launch that Argon's coding was uneven and shaped by benchmark optimization, which Google disputed — a claim neither this week's scores resolves. StarSkirmish's low correlation with the headline indexes could equally mean that long-horizon coding is simply a different skill than the multiple-choice and short-task mix the aggregates reward, not that the aggregates are gamed. And there is a trap in the untuned-benchmark thesis itself: the moment a field decides that practice time against strong opponents is what predicts real performance, that becomes the next thing models are trained to do. Benchmarks do not stay unloved; they get loved to death.
What to watch
Four tells, in order of how much they would change the picture. Whether ARC Prize ever publishes an Argon score at the highest reasoning setting, or whether API instability makes partial-credit results the normal case. Whether StarSkirmish adds a per-model cost table, now that the winning model is the expensive one and second place costs a tenth as much. Whether leaderboard owners start printing harness labels, cost per run and completion rates the way papers print error bars — Epoch and Artificial Analysis both already carry enough metadata to do it. And whether Argon's 18-point variance narrows as Google ships updates, because that spread — not the 97.5 — is the number that decides whether enterprises can plan around the model.
The larger shift is simple to say and hard to absorb: benchmark numbers have stopped being properties of models. For most of the last decade, a score was something you could look up and trust the ranking of. This week, the same model on the same official test was worth 97.5 or 49.8 depending on plumbing nobody publishes, first on one index and eighth on another, a coin flip on its good days. The labs know this, the benchmark owners know it, and the leaderboards still render every number in the same font. Until they don't, the honest reading of any leaderboard is the one Argon's week teaches: name the harness before you name the winner.
