Sakana trains 1000-layer networks with no backpropagation
A paper out of Sakana AI this week takes the oldest objection to local learning and answers it where it bites hardest — depth. Meanwhile the slowdown debate got its most concrete moment yet, on speakerphone.
Sakana AI's PC-ALM is the first layer-local training method shown to work on networks up to 1000 layers deep, according to work published by the lab and posted to arXiv. Predictive coding has long been the credible alternative to backpropagation: each layer nudges its own activity to shrink the prediction error with its neighbours, supervision enters at the output, and the correction creeps backwards through a chain of local compromises. The failure mode is signal decay — in deep, narrow networks the credit signal goes weak long before it reaches the input, which is exactly the regime where backprop earns its keep. PC-ALM's fix is to run primal-dual inference on an augmented Lagrangian instead of gradient flow on an energy. Lagrange multipliers accumulate the local prediction errors, and the authors derive a result that does the work: in a deep linear network those accumulated dual variables recover the exact backprop credit signal, using only communication between adjacent layers. On the benchmark grid from recent predictive-coding scaling work — MLPs at matched width, depth, inference budget and seed, trained for a single epoch on MNIST and Fashion-MNIST — standard predictive coding tracks backprop in wide shallow nets and falls behind as nets get deeper and narrower, while PC-ALM holds close to backprop across the whole grid for all three activations tested, with the largest gains exactly where predictive coding fails and no extra inference cost. It also beat predictive coding on every task tried, including ResNet-18 on CIFAR-10 and Tiny Image Net, and individual neurons settle into damped oscillations during inference. The honest caveat is the task list: these are small vision benchmarks, not language models, and the stated motivation is neuroscience — how a physical system assigns credit for a global objective. Nothing here threatens backprop at frontier scale today. What it does threaten is the assumption that a layer-local rule must be worse at depth, and that is the kind of result that changes what people try next.
President Trump telephoned Nvidia CEO Jensen Huang mid-session at the All-in Summit in Los Angeles on Monday, put the call on speaker, and told the room the backlash to AI and data centers is "all a hoax." Huang asked stagehands for an extra microphone rather than hanging up, and the two spent the exchange agreeing: data centers are "the oil of the next 20, 25 years," opposition to building them is "playing right into the hands" of people who want it stopped, which Trump said could be political opponents or China. "We're not going to let that happen," Trump said. "You're right. We're not going to let that happen, sir," Huang replied, to applause. Trump also joked that Huang can design a chip nobody can copy but cannot work a speakerphone, and that he had "not even known how [Huang] felt about it." Chamath Palihapitiya's post-session verdict was "surreal." The substance behind the spectacle: the summit call came hours after Trump's all-caps post declaring that the only guardrails AI needs are already in his hands, and it puts the world's most valuable company's chief executive publicly on the anti-pacing side of a fight Anthropic started and OpenAI half-joined. This is the enforcement posture we looked at earlier today — Trump's answer to the pacing ask: he already has criminal power over the labs — now with a handshake attached, and an Nvidia representative that did not respond to a request for comment.
What to watch: whether PC-ALM survives a contact with a language model at scale — the reproduction is public, so that test is a GPU allocation away.
Local learning rules have lost to backprop on paper for forty years — is one epoch on MNIST enough to make you rethink your training stack? Tell us in the comments.
Sources: SakanaAI/pc-alm (GitHub)
