Halogen doubles 1M-context decode speed on a 128 GB Strix Halo box
Local inference has been cheap at short context and painful at long context for as long as anyone has measured it. This release attacks that slope directly — on one chip, for one model.
A 128 GB Strix Halo machine running Qwen3.8-Flash-Next now decodes at 38.3 tokens/s with a million tokens of context loaded, up from 27.3 — same box, same session, same prompts, same bits.
The engine is halogen-flash-server, version 0.12.0, written for exactly one GPU (AMD's Ryzen AI Max+ 395) and one model family, with no portability layer and no fallback path. The trigger for the release was public: a reviewer's depth curve on the previous build showed generation falling from 42 tokens/s at 11k context to 16 at 937k, and the maintainer took it as a bug report. The fix was not in the math. At depth, the engine was paying for the step that decides which slices of the context each layer looks at — a serial pass per layer during generation, and a step that re-read the key set repeatedly during prompt processing. Rewriting both, with identical results, moved serial decode from 30.0 to 35.2 tokens/s at 262k context and 18.9 to 32.1 at 1M; with the speculative drafter switched on, 27.2 to 41.7. Cold prefill of a 1,044,480-token prompt dropped from 21.2 minutes to 17.9, about 790 to 937 tokens/s. The 258k rows moved modestly (42.9 to 45.0 tokens/s decode) and the 32k row — the standard served mean the project publishes against rival runtimes — is unchanged.
The numbers come from the project's own harness, so they are vendor benchmarks and deserve the usual discount. What makes this one worth reading is the guarantee attached to them: the new kernels produce output that is byte-identical to the old ones and to plain serial decoding, checked bit-for-bit at 32k, 262k and 1M tokens, with every published quality score unmoved. A speed claim carrying an identity claim is rare — most long-context gains in the wild arrive as a trade, usually against accuracy at depth. This one is sold as pure overhead removal, and the project verified it rather than asserting it.
The caveats are structural, not technical. The engine is not open source, which is the loudest complaint in the thread — one commenter called it "very useful, just needs to be open sourced to be truly excellent." It runs on one piece of silicon. The weights are a 118 GB download, and once loaded the server holds most of the machine: roughly 12 GB free on a 128 GB box, which is why the README tells you not to share the host. This is a single-purpose appliance, not a general runtime. We covered the model itself in August — Alibaba's Qwen3.8-Flash-Next hits Opus-class at one-ninth cost — and the interesting arc since then is that the community's own hardware keeps finding headroom the general-purpose engines leave on the table.
What to watch: whether the project opens the source, and whether llama.cpp derivatives close the long-context gap now that the bottleneck is named in public.
Does a million-token context on a desk-side box change what you'd build locally, or is the memory cost still the real wall? Tell us in the comments.
Sources: halogen-flash-server (GitHub)
