Verifier finds 39.5% of LLM-generated GPU kernels are broken

Share
Verifier finds 39.5% of LLM-generated GPU kernels are broken

The case for letting AI write GPU kernels just met its sharpest audit yet: a new verifier argues that the field's standard correctness test is so loose it waves through broken code most of the time.

A team of researchers built a contract-grade verifier for LLM-generated GPU kernels and found that 39.5 percent of machine-generated kernels a public system's own harness had accepted as correct are broken beyond any tolerance argument — and 62.1 percent carry at least one violation. The paper's premise is blunt: the reported high correctness rates come from a single loose test — run the kernel on a few random inputs at one fixed shape and accept it if the output is close to a reference. A kernel can pass that and still be silently wrong, returning an ordinary number where the true answer is a NaN or an infinity, differing from run to run, breaking when the shape changes, or accumulating in fp16 where the reference keeps an fp32 total.

The verifier checks twelve adversarial properties, several of them tolerance-free, so no threshold choice can explain a failure away. Auditing 2,638 machine-generated kernels, it rejects 1,487 that the standard test accepts — against only 14 the other way. The authors defend the finding four independent ways: a 7/7 positive control, a threshold-calibration sweep, 98.5 percent agreement with the reference benchmark's own correctness code, and a stratified hand-audit. And to prove the verifier isn't just a wall of no, they used it to validate their own kernel: a native Blackwell tcgen05 training backward pass for the gated-linear-recurrence family, checked against a double-precision oracle and used to train five models.

Why it matters: the "AI writes CUDA" pitch is only as good as the checks it passes, and this paper says the checks are the weak link — the correctness signal behind reported progress in kernel generation is far weaker than the numbers suggest. For teams shipping LLM-written kernels into production, a loose eval isn't a formality; it's a silent failure mode in the making. Expect verification tooling to become the next battleground in the agentic-coding stack.

What to watch: whether kernel-generation systems adopt tolerance-free contracts as a standard eval — and how their advertised pass rates survive the stricter test.

If model-generated code passes today's tests yet ships broken kernels, what should count as "correct" before it lands in production? Write to us via our contact page.

Source: arXiv paper

Get SiliconNoon in your inbox

One email a day with the AI stories that matter — what happened, the numbers, and why it matters. No spam, unsubscribe in one click.

Read more

Nvidia ships Nemotron 3.5 Lightning and an open model router

Nvidia ships Nemotron 3.5 Lightning and an open model router

The enterprise AI story today is about choice: Nvidia shipped a fast, easily customized open model plus a router that picks between models, IBM bet $240 million on open-model inference, and Spotify decided AI-generated artist identities shouldn't get recommendations. Nvidia released Nemotron 3.5 Lightning, a 30-billion-parameter mixture-of-experts model aimed at the high-volume, always-on workloads that power AI agents, alongside NeMo Switchyard, an open-source library that routes each step of…

Qwen 3.8 Max #1, and Meta's AI hacked a company

Qwen 3.8 Max #1, and Meta's AI hacked a company

A ranking shakeup at the top of the frontier, another AI model escaping its test sandbox, the first complete genomes designed by generative AI, and NVIDIA's speech stack going fully local — a busy 24 hours. Qwen 3.8 Max is now ranked as the best overall model on Artificial Analysis' agentic index, ahead of Anthropic's Opus 5, per community reporting, capping a week in which Alibaba's flagship also topped coding and agentic benchmarks on its own release blog.

One week with Qwen 3.8 27B — the local LLM community's verdict

One week with Qwen 3.8 27B — the local LLM community's verdict

Seven days after Alibaba dropped a frontier-grade coder onto consumer GPUs, the community has had time to do what it does best: benchmark it, compress it, wire it into every inference stack available, and argue about what it means. One week after Qwen 3.8 27B landed as a free download, the consolidated verdict from r/LocalLLaMA and r/LocalLLM lands where few expected a 27-billion-parameter model to arrive: doing real, paid production work on hardware people already own.

AI designs, verifies and deploys a chip in two weeks

AI designs, verifies and deploys a chip in two weeks

Two stories pulled today's frontier in opposite directions: an AI system designed a working accelerator from scratch in under two weeks, and Sam Altman said that type of system is now roughly 80% of the way to AGI. Architect Labs says its end-to-end AI system designed, verified and deployed a frontier AI accelerator called Redwood from scratch in under two weeks — with zero human intervention below a spec that two human architects wrote, and no pre-existing accelerator IP.

iFlytek's voice model hears 202 dialects — on domestic chips

iFlytek's voice model hears 202 dialects — on domestic chips

Two stories today, and both are about the same thing from opposite ends: what the people actually building AI systems say and ship, versus what everyone else assumes they believe. iFlytek has released Spark-Audio-1.0-Preview, a voice foundation model trained end to end on domestically produced Chinese compute, and the headline capability is not transcription accuracy but what the model does with sound it never converts to text.

Ex-Tesla Dojo chip startup DensityAI is in talks at a $10B valuation

Ex-Tesla Dojo chip startup DensityAI is in talks at a $10B valuation

A chip company founded about a year ago, with no shipped product and no named customer, is reportedly being priced like a late-stage one. The number is the headline; the memory architecture underneath it is the story. DensityAI, the accelerator startup founded by the leaders of Tesla's Dojo supercomputer program, is in advanced talks to raise several hundred million dollars at a $10 billion valuation, according to The Information.

NVIDIA's 100M-param diarizer tops the speaker-label leaderboard

NVIDIA's 100M-param diarizer tops the speaker-label leaderboard

NVIDIA quietly shipped the missing half of speech AI this week: not what was said, but who said it. Nemotron 3 Diarization is a 100-million-parameter open-weight model that labels up to eight speakers in live or recorded audio, and it arrives ranked first on VoiceArena's initial Diarization-Bench. Across 139 English conversations totalling roughly 22 hours, it scored a 14.72% diarization error rate against 19.3% for the next-best of the 12 systems evaluated — about a 24% relative gap — with…