AI inference is redrawing the storage hierarchy

Share
AI inference is redrawing the storage hierarchy

The money has followed GPUs for years, but at GMIF 2026 in Shenzhen the argument was about everything behind them — and the numbers backing it are getting hard to ignore.

China's 140 trillion daily token calls are forcing a redesign of the storage stack. Now in its fifth year, the Global Memory Innovation Forum gathered storage makers, analysts and chip designers around one theme: inference, not training, is now the workload that dictates hardware. China's daily token calls had already passed 140 trillion as of March 2026 — a figure confirmed in official government data, and consistent with the growth curve we tracked in China expects to burn 100 quadrillion tokens this year — and Samsung Memory CTO Kevin Yoon put the scale in physical terms from the GMIF stage: that daily volume converts to more than 20 times the entire collection of China's National Library in tokens, and close to six times every book humans have ever published. A Morgan Stanley executive director told the audience that memory and storage already account for nearly half of 2026 cloud data-center capital expenditure excluding HBM, rising to 53% next year. The implication is that storage is no longer the boring line item.

The talk of the conference was how the old rules of the storage pyramid break under agentic workloads. Solidigm's Asia-Pacific sales lead described the shift: when storage was cheap, the industry collapsed hierarchy layers; now AI data costs are forcing the pyramid to be subdivided again, and North American hyperscalers have started swapping high-capacity QLC SSDs into roles HDDs used to own. SanDisk framed the new split by distance from compute — SSDs wired directly to GPUs feed hot KV cache, network-attached SSDs serve the same compute at larger scale, and capacity-tier drives absorb data nobody touches often. Silicon Motion went further, showing controllers that classify a workload by phase — planning wants low latency, retrieval wants throughput, execution wants bandwidth — and allocate QoS within one drive accordingly.

Underneath, the NAND medium itself is stretching. Samsung's Z-NAND keeps the OS and KV cache in DRAM while parking read-heavy model weights on SLC-based dies, an architecture aimed at the DRAM cost pressure of on-device inference. Peking University's IC school outlined two paths to high-bandwidth flash — shrinking arrays for latency, or adding interfaces for bandwidth — while warning that both still fight density tradeoffs and write endurance that KV cache updates stress directly. His lab and 燕芯微 are also exploring computing directly inside NAND arrays. Meanwhile 联芸 and 寅谱's joint AI SSD prefetches the next likely MoE experts from SSD into memory before the router asks for them, and 佰维存储's chairman summed up the mood: data supply efficiency now determines compute output.

What to watch: whether HBF prototypes reach roadmaps before QLC-eats-HDD procurement does the near-term work.

Is storage about to become the next AI bottleneck — or just the next AI pricing opportunity? Write to us via our contact page.

Get SiliconNoon in your inbox

One email a day with the AI stories that matter — what happened, the numbers, and why it matters. No spam, unsubscribe in one click.

Read more

Broadcom lines up over $50B to finance OpenAI's custom chip

Broadcom lines up over $50B to finance OpenAI's custom chip

The AI buildout is increasingly being paid for with borrowed money — and today's biggest example puts OpenAI's in-house silicon at the center of it. Also: Nous Research banks a $1.5 billion valuation, and an AI-built Adobe clone suite declares "software is over." Broadcom has been working to arrange more than $50 billion in financing for OpenAI's custom AI chip, with Oracle in separate talks on financing for a large chip purchase, the Wall Street Journal reported.

Musk: only three organizations have enough coding data

Musk: only three organizations have enough coding data

The chip suppliers and the biggest buyer of chips are suddenly telling the same story: capability is no longer the bottleneck — money is. Elon Musk spelled out what he thinks the scarcest training input actually is, and the chiefs of Intel and Broadcom spent a day in Singapore arguing that access to capital may decide who survives the next phase. Elon Musk says only three organizations in the world have accumulated enough coding training data — and xAI was not one of them.

Chip stocks slide after Broadcom chief calls AI investing a casino

Chip stocks slide after Broadcom chief calls AI investing a casino

A single interview line hit the tape overnight and global chip stocks sold off hard — the latest sign that AI capex sentiment is much thinner than the index highs suggest. Broadcom CEO Hock Tan compared parts of AI investing to gambling in a casino, and semiconductor stocks worldwide took the cue. According to the Financial Times, Tan's remarks — likening some AI investment to betting at a casino — rippled through markets already nervous about the pace of AI spending.

ASML's order book stretches to 2030 as profit beats estimates

ASML's order book stretches to 2030 as profit beats estimates

ASML is the toll booth every leading-edge AI chip has to pass through, and its third-quarter report says the queue in front of it is getting longer, not shorter. ASML posted third-quarter net sales of €11.66 billion — a hair under the €11.72 billion analysts expected — but net profit of €4.96 billion against a €4.1 billion consensus, and told investors that customers have started asking about orders for 2029 and even 2030. Gross margin landed at 57.5% versus 56.1% expected.