Gemini 4 Argon played StarCraft — and the frontier's still ahead
A benchmark nobody optimizes for, and a benchmark everybody does. Both point the same direction: Google is closer to the frontier than it has been since February, and still not on it.
On a benchmark where each model gets exactly one hour to write a StarCraft bot, Gemini 4 Argon finished third — functionally tied with Claude Opus 5.5 and Claude Fable 5.1, and behind two OpenAI models. StarSkirmish Bench v0.2 runs each lab's flagship in that lab's own command-line harness at its highest reasoning setting, hands it an hour, and then plays the resulting Protoss bots against thirteen human-written and demo bots. Argon scored 37 to 39 across five runs; GPT-6 Astra and GPT-6.1 Sol took the top two spots. The gap that should worry Google is not the podium — it's the ceiling. Against just the three strongest human-written bots, the best model in the field, GPT-6 Astra, won 7 percent of its games. Argon also varies far more run to run than its rivals: an 18-point spread between its best and worst hour, against 9 points for Astra and 8 for Opus 5.5. That variance is the number worth watching, because it says the model's performance on a long-horizon agentic task is still partly a coin flip.
The benchmark's own authors make the more interesting claim. StarSkirmish no longer tracks the most-reported evals — it barely correlates with the Artificial Analysis Intelligence Index, and it has drifted away from Terminal-Bench 4.0, which it tracked closely in v0.1. It still moves with Vals' IOI and WeirdML. One reading, theirs: the heavily benchmarked evals have become targets models are tuned for, while the unglamorous long-horizon ones still measure something. The strongest single predictor of a model's score here was how much it practiced against the seven best bots during its hour — even while it was losing to them.
Read Argon's benchmark week properly and it becomes a story about harnesses, not capability. The number everyone quoted this week — 97.5 percent on ARC-AGI-3 at $4,800 per run, the second lab after OpenAI to clear 95 percent — comes from ARC Prize's Provider Adapter Harness, which lets a model keep its reasoning across actions and compact its own context between them. Run the same model on the Standard Harness, which allows only simple notes between rounds, and the score falls to 49.8 percent. ARC Prize also could not complete the full 55-game set at low and medium reasoning because of API problems, and published no score at all at the highest reasoning level; the reported figures include partial credit from aborted runs. Epoch AI's Capabilities Index puts Argon at 165 — behind Opus 5.5 at 167 and GPT-6 Astra at 166, ahead of Gemini 3.8 Flash at 157 — but sixth on software engineering and eighth on math. We flagged the real-world coding doubts when the model shipped — Google employees question Gemini 4 Argon's real-world coding — and neither of this week's numbers settles them.
What to watch: whether anyone reruns StarSkirmish with a per-model cost table, now that the winning model is not the cheap one.
Which number would you trust to tell you a model can actually do a day of work — the benchmark score, or the hour it spent losing practice games? Tell us in the comments.
