The CPU is back — Red Hat's case for CPU-first inference
Red Hat's engineers argue the agentic era is rewriting the compute economics of AI — and the data from Intel, Arm, and NVIDIA backs them up.
Red Hat is making a forceful case that the CPU is back in AI infrastructure, and it's hard to argue with the numbers it marshals. In a post from its developer advocates that hit the top of Hacker News, the company argues that agentic AI — tool calls, multistep reasoning, and orchestration across many small specialized models — fundamentally changes where inference compute should live. Intel's data quantifies the shift: the CPU-to-GPU ratio in data centers is moving from roughly 1:8 in training to about 1:4 in inference, and converging toward 1:1 in agentic workloads, with some customers already reporting four CPUs per GPU. Research from Georgia Tech and Intel found CPU-side tool processing accounts for 50–90% of end-to-end latency in agentic workloads — meaning the "thinking" around the model now costs more than the model itself.
The hardware industry has clearly noticed. NVIDIA launched its Vera CPU in March as "the world's first processor purpose-built for agentic AI," with first deliveries going to Anthropic, OpenAI, and Oracle Cloud Infrastructure; its Vera Rubin NVL72 rack pairs 72 GPUs with 36 CPUs, a deliberate pivot from the old 1:8 ratio. Arm's AGI CPU arrived the same month, and CEO Rene Haas estimates agentic data centers will need 120 million CPU cores per gigawatt versus 30 million today. OpenAI and AWS's $38 billion infrastructure partnership explicitly includes headroom to scale to tens of millions of CPUs. Morgan Stanley puts the incremental CPU market from agentic AI at $32.5–$60 billion by 2030.
None of this means the GPU era is over — Red Hat is explicit that high-concurrency serving of large models remains GPU territory, and a single H200 still crushes any CPU VM on raw generation throughput. But the software stack has caught up fast: vLLM's CPU backend now supports continuous batching, PagedAttention, and prefix caching across x86, Arm, and even Apple Silicon, and Red Hat open-sourced a benchmarking framework (vllm-cpu-perf-eval) to bring reproducibility to a space that's been a mess of vendor numbers. The practical takeaway for teams building AI products: if your workload is agentic, orchestration-heavy, or small-model-at-the-edge, the cheapest GPU you can buy may be the CPU you already own.
What to watch: whether vLLM's CPU backend holds up on decode-heavy workloads beyond single-digit concurrency — the ceiling that skeptics say still separates CPUs from HBM-backed GPUs.
If agents really are going to need 4 CPUs per GPU, whose data center bill changes the most — the hyperscalers', or the enterprises running agents on-prem? Tell us in the comments.
