Hugging Face runs async RL training with no NCCL and no weight ship
The hardest part of reinforcement learning is not the algorithm. It is getting a trainer and a bunch of inference servers to keep passing weights back and forth without one of them stalling. Hugging Face just published a recipe that does it with three rented jobs, one storage bucket, an HTTP proxy, and no NCCL at all.
Hugging Face trained a reasoning model end to end using async GRPO and LoRA, and the whole weight sync is a file upload. The setup is deliberately small: one job runs the trainer on DeepSeek-R1-Distill-Qwen-1.5B, two more serve vLLM replicas, and a Hub storage bucket holds checkpoints. Because the policy is trained through a rank-1 LoRA adapter, the thing that has to travel to the inference servers after an update is a few megabytes instead of the roughly 3 GB full model. vLLM can hold several adapters loaded at once, so rollouts already in flight finish against the policy they started with while new ones pick up the latest weights. Nothing is lost when a job gets preempted either — the adapter lives in the bucket, so the trainer resumes where it stopped. The work landed in TRL's async GRPO trainer. The team's own dashboard shows the reward curve climbing over the run, all 252 adapter loads succeeding across 126 syncs and two replicas, and 64,728 rollouts crossing the public proxy without a single transport error.
The interesting part is what broke when they optimized. The first configuration had the trainer as the bottleneck, not generation: the rollout queue sat full, and the second replica was decorative. Fixing the training step lifted generation throughput from 4,600 to 25,000 tokens per second without touching vLLM at all. Then weight sync — 7.6 seconds every four steps — swallowed a quarter of wall-clock time, and adding a third inference replica moved throughput from 25,000 to 26,000 tokens per second, which is to say nowhere. The real limit turned out to be a conservative client-side cap on how many requests could be in flight at once. Raising it to 384 made training the bottleneck again, which is the correct problem to have. Net result after five experiments: 3.9× faster, 31 percent more samples trained on, and basically the same reward curve. There is also a routing lesson worth pairing with our KV-cache policy test: the proxy sends each prompt to the replica most likely to already hold it in cache, hitting 84.5 percent affinity against a theoretical floor of 12.5 percent cold requests — because every prompt in a run starts with the same chat template and system prompt, matching on shared prefixes tells you nothing about where a specific prompt actually lives.
What to watch: whether this adapter-only pattern reaches bigger models, where a few megabytes becomes a few hundred and HTTP stops being a reasonable transport.
Would you run an RL loop on rented, preemptible GPUs if a checkpoint could never be lost — or is that still too much faith in a file upload? Tell us in the comments.
Sources: TRL pull request #7017 (GitHub)
