Deep Dive — Memento 3 finishes a benchmark that was already finished
A team at University College London and Huawei's Noah's Ark Lab has taken ARC-AGI-3's public game set to the ceiling: every level of all 25 games, a mean Relative Human Action Efficiency (RHAE) of 100.0, and a total action count of 7,518 — 44 percent of the 17,135 actions first-time human players needed. The system that did it, described in a paper posted October 8, never updated a single model weight. Its backbone, Claude Opus 5, was frozen; its reasoning effort was set to extra-high; its harness was a modified copy of the community's own best-scoring open entry. What changed between an official run of Claude Opus 5 on this benchmark — 59.3 RHAE points lower, by the paper's own comparison — and a perfect score is a Markdown file and a Python program the agent writes about the game it is playing.
That framing is the point. ARC-AGI-3 was built as the benchmark that measures whether an agent can learn inside an unfamiliar world: no instructions, no natural-language goals, rules you have to discover by poking at them. It launched in March 2026 with humans clearing 100 percent of environments and frontier models under 1 percent. Seven months later the public set has been reported perfect by a research lab, a university group, an independent developer and two hardware companies' teams in succession, ARC Prize has already restructured the next generation of the exam, and the interesting competition has moved from whether you can clear the board to how few moves it takes, how many tokens it burns, and whether any of it transfers past a deterministic pixel world.
Code as Model
The mechanism the UCL and Huawei researchers call Code as Model has two artifacts, both living outside the model. The first is a rulebook: a natural-language Markdown document that records the agent's current hypotheses about the environment — what objects exist, what each action does, what the goal appears to be, and explicitly what is still unknown. The second is an executable world model: Python code compiled from that rulebook, which a planner queries to search for action sequences before spending any real moves.
The loop around them runs as observation, reflection, rule revision, compilation, verification. When reality disagrees with the model, the agent amends the rulebook and recompiles — the paper's running example being a puzzle where the agent first believes a tile flips the carried pattern by mirror depending on entry direction, then discovers on testing that it rotates clockwise by 90 degrees regardless of direction, and rewrites both the prose rule and the code branch it maps to. An update is only accepted when the LLM judges the code faithful to the rulebook and a cell-by-cell replay of the 64×64 game grid reproduces the transitions actually observed. Rejected code does not enter the model.
Two supporting details do more work than they look like they do. Every revision is committed to a Git repository with the iteration number tagged, so the agent can diff two versions of its own beliefs, recover a hypothesis it discarded, or roll back an edit that broke a working executable — version control as learning state, not just housekeeping. And because a finite history never uniquely determines the true rules, the system can run a population of two rulebook–executable pairs in parallel, each starting from different hypotheses, sharing every observation: on wa30, the game with the largest human action budget, keeping two models cut scored actions from 899 to 597, improving or matching efficiency on eight of nine levels.
The ablations are honest about where the win lives. Removing the rulebook while keeping everything else byte-identical still reaches RHAE 100 on the three test games — it just gets there more slowly, 677 actions instead of 617 and 830 agent turns instead of 678, a 9 percent action penalty and 18 percent more turns. The rulebook is a convergence accelerator, not the source of the ceiling. With a strong enough model, the score was reachable either way; the book is what makes it cheap.
When the model can go home
The paper's second experiment is the one to keep. The team pointed the same machinery at Atari Pong, where the ball keeps moving and the rules have to be inferred from physics rather than menu text. Early on, the agent's rulebook about paddle bounces contradicts an observed trajectory, gets revised, and converges into a feedback controller. Once the controller wins 21–0, the LLM is removed from the loop entirely: three evaluation episodes with three different openings, three shutouts, zero model calls during play.
The cost comparison in the paper's case-study table is the sharpest number in it. The learned controller reached maximum score after 9,504 emulator frames. The model-free reinforcement learning baselines listed alongside it ran to roughly 200 million frames. And the control condition — the same LLM, one call per frame, no world model, no memory between calls — finished at minus 19, barely distinguishable from a uniformly random policy at minus 20.7. A frontier model reacting to pixels in real time is worthless at this game; the same model that first wrote down how the game works and then stepped away is perfect at it. That gap is the entire argument for externalized world models, stated in one row of a table.
A leaderboard nobody can win twice
Memento 3's total of 7,518 actions beats the previous community-leaderboard finisher, the independent developer Sergey Rodionov's baseline1, which cleared all 25 games in 8,347 actions for a mean RHAE of 99.0 — and baseline1's codebase is the starting point Memento 3 was built on, so this is an iteration on the record, not a demolition of it. The rest of the paper's comparison table reads as a scale of how far harness work has come: Continual Harness from Princeton at 20.5, DreamTeam at 38.1, NVIDIA-affiliated NOOA at 85.1, OPINE-World at 78.4, all measured from official ARC Prize scorecards.
It is also, importantly, not the only perfect public-set report on record. NVIDIA's AVO agent cleared every ARC-AGI-3 public level back in August with its own count of 6,624 actions — a cross-system comparison NVIDIA itself flagged as uncontrolled — and Kaiming He's harness gives Claude a perfect ARC-AGI-3 score covered the VISTA group's October 1 result, also 100.00 RHAE, with 7,302 actions against the same 17,135 human baseline, alongside the program-synthesis entry Tycho's 100.00. Different papers count actions under different conditions, which is exactly why the official base-model leaderboard still separates Standard-harness from Provider-Adapter results rather than crowning one number: OpenAI's GPT-6.1 Sol, the top entry there, posts 52.7 percent under the provider-neutral interface and 96.2 percent under the adapter that preserves its reasoning state.
The bigger context arrived last month: ARC Prize concluded its own exam was finished. GPT-6 saturated ARC-AGI-3. The next benchmark grades invention covered the announcement that ARC-AGI-4 becomes a benchmark for autonomous open-ended innovation, after GPT-6 Astra hit 99.9 percent on the semi-private set under the adapter harness while scoring 62.7 percent under the standard one — a 37-point gap between two ways of asking the same model the same questions. ARC Prize's own line still anchors the series: as long as a measurable gap between human and machine learning efficiency remains, there is no AGI. What Memento 3's arrival shows is that on the public set, that gap is now closed by writeable, debuggable files around a model that never moves.
What skeptics will say
The caveats start with the set itself. The public games are ARC Prize's front door — built for human play, and the most exposed to memorization as the benchmark's profile has risen. The semi-private and private competition sets remain the unsaturated territory, and no perfect public score answers what happens there. The paper's own ablation invites a second reading: if a run without the rulebook still scores 100, then part of what is being measured is convergence cost on small, deterministic, gridworld-style games, and the paper is upfront that whether the method survives messier, less rule-governed environments is untested.
The cost question sits underneath everything. Every ablation run was expensive enough that the authors selected only three games for it, calling the time and monetary cost substantial — and then reporting no dollar or token figures at all, on a backbone running at extra-high reasoning effort inside a full agentic coding harness. Action counts are a clean, comparable currency; they are not the bill. And "recursive self-improvement" deserves its usual dose of skepticism: the LLM improves nothing about itself here. It maintains inspectable artifacts that make its next interaction cheaper — a real and useful result, but the RSI label is a research lens on external memory, not a model that designs its own successor.
What to watch
The near-term tells are concrete. Whether ARC Prize formally scores Memento 3's run alongside AVO's and VISTA's under one protocol, since right now the perfect scores circulate as separate reports with separate counting rules. Whether any of these teams publish cost per run, which would turn the harness race from an action-count contest into an economic one — the same question the field's harness-tax studies have been forcing onto coding agents. Whether the rulebook-plus-executable pattern shows up outside game grids, where the environment does not replay itself cell by cell for you. And ARC-AGI-4's actual design: if invention really replaces interactive learning as the measuring stick, the artifact-building instinct Memento 3 demonstrates — write the world down, compile it, check it against reality, keep the history — is the closest thing we currently have to a template for a benchmark that grades it.
