Loop a transformer twice, save 18% of training compute
Two stories from the research world today: an architecture paper that puts hard numbers on one of this year's more interesting ideas in model design, and a peer-review system that is quietly rewriting the economics of who gets to submit.
Re-running a transformer's middle layers twice can cut training compute by up to 18% — if you pay for the extra passes honestly. That's the finding of SMELT (Sparse MoE Transformer, middle layers Loop Twice), a paper from teams at Tsinghua University, ByteDance Seed, M-A-P, and TokenWave.AI. The authors' starting point is a complaint about prior looped-transformer work: most of it compared models on parameter counts while ignoring that looping costs extra FLOPs at train and inference time. SMELT locks three budgets instead — per-token compute, total parameters, and KV cache — then uses MoE to square the circle: narrow the hidden dimension to pay for the second pass, and enlarge the expert pool to restore parameter count.
The recipe that survived the ablations: loop only the middle half of the layers, loop them exactly twice (three or four passes hurt), and build the model deeper-narrower than a standard one. Tested across 32 models from 100 million to 54 billion total parameters, the scheme saved 6.8–10.0% of training compute at a 10²⁰-FLOP budget, widening to 14.7–18.0% at 10²¹ — and the fitted scaling laws show the gap grows with scale, not shrinks. The mechanistic analysis is the best part: on the second pass, attention sinks at the first token collapse from 0.60 to 0.02 of attention weight, freeing that budget for content the model actually needs — the second loop refines the first pass rather than redoing it. The gains concentrate where recurrence should help most: code (+20.4% compute efficiency), math and finance, and long-context and few-shot tasks. It's a concrete mechanism behind the recurrent-depth designs shipping in frontier models like Astra, and (if you need the base layer) we explained the substrate in AI 101 — What is a transformer?.
ACL proposes tying paper submissions to reviewer capacity — with a lottery for the rest. The Association for Computational Linguistics announced a "Sustainable Reviewing Policy" for its ARR submission cycles in which each paper must "pay" for itself by supplying a qualified service contributor — a reviewer or area chair — or else enter a lottery for whatever spare review capacity remains. Authors are capped at 20 total submissions and 5 first-author submissions per cycle, non-author contributors can be nominated but must vouch for the work arXiv-endorsement style, and accounts that systematically abuse the system face penalties or bans. It is the first major AI venue to hard-link submission volume to reviewing labor, and the r/MachineLearning reaction splits between "long overdue" and complaints that the caps are still too generous. Expect other conferences to watch this one closely — reviewing capacity is the binding constraint everywhere.
If submitting a paper meant reviewing two, would that fix peer review or just lock newcomers out? Tell us in the comments.
