UrbanGround puts AI agents in a full-scale 3D Hong Kong
A new sandbox drops multimodal models into a photoreal, physically simulated replica of Hong Kong and asks them to do something benchmarks rarely test: walk somewhere. They can describe the street. They can't reliably get across town.
UrbanGround, from a team spanning Shanghai Jiao Tong University, the National University of Singapore and several Chinese labs, is a real-scale urban sandbox built from Hong Kong's territory-wide 3D geospatial data — the Lands Department's 3D visualisation map and 3D pedestrian network, rendered in Unity as a continuous first-person environment. An agent sees only what a person walking the street would see: RGB frames from its own camera, an interactive map with no route computation, and a small action space of movement, sprinting, jumping and camera turns. No hidden coordinates, no distance-remaining readout, no privileged simulator state. Thirteen task types are arranged across five capability levels, from visual recognition and orientation up through multi-stop route planning and replanning after a road closes mid-journey. The simulator and the model run as separate processes, so any multimodal model or agent framework can drive the same body.
The findings are a clean picture of where spatial intelligence actually breaks. Models do fine recognising what's in front of them and answering local spatial questions, and some manage short routes when the destination stays visible. Performance falls off a cliff the moment the goal has to be inferred or held over a long trajectory: routes that require inferring a destination from an instruction are no longer than short-range ones, yet scores drop substantially. The dominant long-navigation failure isn't an inability to start moving — it's unstable progress, where a model approaches the goal and then regresses, or stops short outside the arrival radius, never verifying it arrived.
Two details are worth more than the headline numbers. First, agents cheat: the paper measures how much of an episode is spent on registered pedestrian routes, and adherence falls far short of complete compliance even on local tasks — models cut a shorter path to the evidence they need and sacrifice road compliance to answer faster. Real-world movement constraints simply aren't held as a persistent part of spatial reasoning when they compete with finishing the task. Second, novelty kills: when a road closure is announced mid-episode, route-recovery success stays low across models even though pedestrian-network adherence stays high. The agents keep producing locally compliant movement without ever rebuilding the plan.
The generational gap is also more specific than "newer is better." GPT-5.2 and Gemini-3.1-Pro stay close to recent models on visual recognition, but fall markedly behind on orientation understanding and active exploration — recent advances have mainly improved preserving directional evidence. And counterintuitively, agents do worse at dusk than at night, likely because dusk lighting is harder to read than darkness.
The take: urban agency isn't a perception problem that better vision will solve. It's a persistence problem — holding a spatial frame across hundreds of steps and revising it when the city changes. Benchmarks that grade single frames will keep reporting progress that doesn't survive a walk around the block.
Would you trust a model that can name every shopfront but can't find its way back after a detour? Tell us in the comments.
Sources: UrbanGround (GitHub)
