Google open-sources TorchTPU: native PyTorch on TPUs, no rewrite

For a decade Google sold TPUs on price per FLOP and treated the software gap as somebody else's problem. On Tuesday it stopped: TorchTPU, the native PyTorch backend for its chips, is open source — and DeepSeek spent the same day giving away its serving stack.

Google released TorchTPU as an open-source project at PyTorch Conference 2026, and the pitch is blunt — existing PyTorch code runs on a TPU without a rewrite. The stack lives in Google's PyTorch org on GitHub, was co-developed with the PyTorch team at Meta, and replaces the route TPU-curious developers have had to take for years: PyTorch/XLA, with its lazy tensors, mystery recompiles and debugging that never quite lined up with the code on screen. In Google's announcement, Mark Lohmeyer and Bill Jia describe the outcome as TPUs joining GPUs and CPUs as "a native option for PyTorch" — train, serve and deploy with the tools teams already run, including vLLM, SGLang, Ray, TorchTitan and Hugging Face.

The mechanics matter more than the marketing. You debug in eager mode, then hand the same model to torch.compile for throughput and lower memory overhead, with Pallas kernels and distributed primitives built in rather than bolted on. Meta is the loudest endorser because it has the most to gain from not being single-vendor: Jana van Greunen, who runs PyTorch engineering there, said Meta "cannot be constrained by a single hardware architecture," and that TorchTPU lets it push LLM, ads and recommendation workloads onto Google silicon — and feed bugs back to both engineering teams.

The integrations are where this stops being a science project. Google shipped a vLLM backend built with Inferact, keeping upstream worker models, schedulers and API servers intact so existing inference stacks move over without being rewritten, with continuous batching, quantization and multi-chip parallelism on TPU hardware. SGLang got a native backend via RadixArk that reuses KV cache across requests. llm-d hit 1.0 with Red Hat, IBM and more than 500 contributors across 25 organizations, adding prefix-cache-aware routing, prefill/decode separation and session-aware routing for agents — and on Google Kubernetes Engine it lets one endpoint dispatch across GPUs or TPUs, which is the version enterprises actually want, because it means a migration is a scheduling decision rather than a re-platforming project.

Training is the other half, and the numbers there are the ones to hold onto. TorchTitan, the PyTorch-native reference trainer, reached up to 52% model FLOPs utilization on Qwen 3 8B while scaling to 8,192 Ironwood chips (TPU v7x), per Google, with recipes covering Llama 3 70B and 405B, Qwen 3 235B in a mixture-of-experts configuration, and FLUX.1 12B. VeRL got an upstreamed TPU backend and a GRPO reference recipe, so reinforcement-learning post-training is in preview on the same silicon. Ray now treats TPUs as a first-class accelerator. LiteRT, the on-device runtime formerly known as TensorFlow Lite, joined the PyTorch ecosystem, so models converted once can run on phones, laptops, browsers and IoT hardware.

Independent evidence that it works comes from Hugging Face, which spent the launch running its own test suite rather than repeating the press release. In its write-up, the team reports 96.5% of the Transformers tests that run on TPU now pass, 34 issues filed against TorchTPU and 19 already fixed — and, more tellingly, that TPU runs exposed bugs in Transformers itself, in StaticCache and the Qwen3 tensor-parallel plans, which were fixed upstream. A TPU-tuned ragged-dot kernel from the Hub made decoding 21.5× faster on Qwen3-30B-A3B and 12.8× faster on Gemma 4 26B-A4B at batch size 64. Diffusion pipelines run natively too: on a Cloud TPU v6e-4, a 1024×1024 Qwen-Image generation took 3.5 seconds compiled against 8.0 seconds eager.

Context for why this week: Amazon released TorchNeuron, its equivalent for Trainium, one day earlier, and is upstreaming code that decouples PyTorch features from Nvidia's hardware. TechCrunch's Julie Bort frames both moves as an assault on Nvidia's real stronghold — not the chips, but the fact that AI apps are written to run on them by default, with PyTorch used by roughly 85% of AI applications. Google is trying to one-up Amazon on breadth: it worked directly with Meta, Hugging Face, Inferact and RadixArk so its libraries behave like upstream PyTorch, and it added a TPU agent suite to automate onboarding, diagnostics and inference tuning for teams porting models over.

Here is the part the announcements skip. Portability is not the same as parity. Nvidia's advantage was never the PyTorch API — it is the decade of hand-tuned CUDA kernels underneath it, and 96.5% of a test suite passing is a good launch number, not a production guarantee. The other constraint is supply: Ironwood capacity is not something a startup can rent by the hour the way it rents H100s from a dozen neoclouds, and if you cannot get chips, an easier port does not help you. Google's honest claim is narrower than the headlines: it has removed the excuse, and it has made the price-per-token comparison a real comparison instead of a thought experiment.

DeepSeek open-sourced DeepInfer, the inference framework that serves V4.1-Flash, under Apache 2.0 — the production stack, not the weights. It builds on vLLM with tuned compute kernels, distributed inference and speculative decoding, ships alongside deepseek-recipe so self-hosted deployments process tokens the way the official API does, and DeepSeek says code for future models will land in the same repository, per the DeepInfer repo. The company's stated aim is to "lower compute costs, reduce energy use, and make AI more affordable" — which is also, conveniently, a price ceiling for every lab selling tokens.

Two details are worth more than the announcement. The New Stack notes the release gives Huawei's chips equal billing with Nvidia, which is the line Western infrastructure teams will read twice: DeepSeek is publishing a serving path that is not Nvidia-only at the exact moment export controls are the live policy question. And the timing is genuinely odd — the same week Homeland Security excluded DeepSeek from federal agencies, a story we covered in DeepSeek banned from federal agencies. Its weights aren't. The weights were already out. The serving stack was the piece that kept "free model, expensive inference" a comfortable moat for US labs, and that piece is now public.

The counter-argument is that this is a gift with a strategy attached: DeepSeek's business is API tokens, and making self-hosting cheaper looks like giving away the margin. But the direct effect is that a model already priced at 20 cents a task gets cheaper to run on your own hardware, and every provider quoting a per-token price has to answer for it. Nathan Lambert put the strategic read plainly on X — "a wild way to get adoption on your models" — and noted the government ban landed in the same news cycle.

What to watch: whether Nvidia answers with a PyTorch-side move of its own, and whether Hugging Face's 96.5% pass rate holds as Transformers v5 goes from launch to general availability.

If PyTorch runs the same on every accelerator, what exactly are you buying when you pick a cloud? Write to us via our contact page.