Xiaomi's MiLM Plus ships PROVE, a benchmark for AI object removal
The AI-editing arms race just got a proper measuring stick: Xiaomi's research team shipped an evaluation framework that says the metrics everyone uses to judge object-removal models are out of step with human perception. Meanwhile, the open-source fine-tuning favorite Unsloth put local model training on the desktop.
Xiaomi's MiLM Plus research group released PROVE, an open evaluation framework for object removal in images and video that its authors say fixes what standard metrics get wrong. The work, accepted at ACM Multimedia 2026, pairs two new perception-aligned metrics — RC-S for spatial coherence and RC-T for temporal consistency — with PROVE-Bench, a real-world benchmark of 180 videos split between a paired dataset with motion augmentation and a harder 100-video subset with no ground truth. Code, datasets and a project page are all public.
The problem PROVE targets is structural, not cosmetic. Object removal is a one-to-many task — many plausible restorations exist for any erased region — so reference-based scores like PSNR, SSIM and LPIPS reward models that play it safe, even copying pixels, instead of erasing convincingly. No-reference metrics have their own blind spots, systematically favoring blurry output, and global temporal scores miss localized flicker in exactly the edited region where these models most often fail. RC-S and RC-T attack that by scoring only the edited area: RC-S compares feature statistics between the removed region and its surroundings, RC-T tracks restored regions across adjacent frames, and neither requires a reference video. Across image and video benchmarks, Xiaomi reports the new metrics align substantially better with human judgments than existing protocols.
The take: evaluation is generative media's weak link — model leaderboards are only as trustworthy as the metrics underneath them, and published PSNR wins have been known to contradict what viewers actually see. PROVE is also a signal that Xiaomi's AI arm, which built the MiMo line that has topped open-model leaderboards this year, is doing serious research-grade work rather than just shipping phone features. Whether RC catches on as the default way to grade editing models is now the interesting question — a metric that needs no ground truth is far easier for the field to actually adopt.
Unsloth, the open-source project best known for making model fine-tuning dramatically faster, launched Unsloth Desktop — a free desktop app for running and training local models on Mac, Windows and Linux. The project bills it as the first desktop app that both runs and trains models, bundling pre-compiled binaries so setup is a download rather than a dependency hunt. It closes a real gap: local-AI users have had chat-style apps on one side and command-line training on the other, with nothing comfortable in between. For hobbyists and small teams, that means fine-tuning their own model on a laptop is now a GUI operation, which is exactly the kind of barrier-lowering move that grows the open-weights ecosystem.
What to watch: whether PROVE's reference-free metrics become the default way to grade editing models — and whether some vaunted PSNR wins start looking different under them.
Metrics decide which models win, and most editing benchmarks are broken. Should the field retire PSNR-style scores for perception-aligned ones? Tell us in the comments.
Sources: PROVE (GitHub) · unslothai/unsloth releases (GitHub)
