Hugging Face Split the Nodes, Dropped NCCL, and Cut Async GRPO Training From 3h 27m to 53m

Four Hugging Face engineers published "Async GRPO with LoRA across HF Jobs" on September 10, 2026, reporting that a reinforcement learning setup with the trainer and the inference servers on separate machines and no NCCL at all finished 500 training steps in 53 minutes instead of 3 hours 27 minutes. The enabling device is a rank-1 LoRA adapter that weighs a few megabytes against roughly 3 GB for the full 1.5B model, small enough to travel as a file through a Storage Bucket. ASAP works only from the five logged experiments and instrumented metrics in that post to trace where the bottleneck moved.

A rank-1 adapter of a few megabytes is what makes the whole arrangement possible

The case for LoRA in reinforcement learning rests on how much signal each step carries. Thinking Machines Lab argued in "LoRA Without Regret," published in September 2025, that LoRA matches full fine-tuning for policy-gradient RL even at rank 1, because the advantage function supplies only about 1 bit of information per episode. With that little to absorb, a rank-1 adapter has enough capacity.

The Hugging Face post draws out the systems consequence. A rank-1 adapter for a 1.5B model is a few megabytes while the full model is around 3 GB, so a policy update can ship the adapter instead of pushing gigabytes to the inference workers. vLLM can also hold several adapters loaded at once, so rollouts already in flight finish under the policy they started with while new rollouts pick up the latest one.

The constraint came from the structure of Hugging Face Jobs. One Job is one container on one VM, capped at 8xH200 per node. A single Job cannot spawn multiple nodes to hold a trainer plus a fleet of vLLM servers, and Jobs cannot communicate across nodes. There is no shared local disk and no shared localhost. With full-weight sync, the arrangement ends there.

The answer arrived through the filesystem. Hugging Face Jobs provide volumes backed by Storage Buckets, mounted as a FUSE filesystem in every Job. The trainer saves the adapter every few optimizer steps, publishes the directory with an atomic rename, and sends the path to vLLM's /v1/load_lora_adapter endpoint. Because that endpoint takes a path rather than tensors, the two processes only need to see the same path. Nothing in TRL or vLLM had to change.

The full setup is three Jobs, one bucket, and one proxy

The layout is a trainer Job running AsyncGRPOTrainer with LoRA and FSDP, two vLLM Jobs serving the base model plus whatever adapter the trainer last published, one Storage Bucket mounted at the same absolute path in all three, and a small proxy on the trainer Job at 127.0.0.1:8000. The trainer uses an h200x2 Job and each replica uses a single H200, and running all three costs about 20 dollars per hour.

The number of adapter slots follows from max_staleness. Every weight sync bumps the policy version by one, and max_staleness sets how many versions a rollout sample may lag before the trainer discards it. At 4, vLLM must serve the current policy plus the four before it, and each swap needs one more slot while the new version loads before the oldest unloads. That is why the flag reads --max-loras 6. With only five, vLLM would silently evict a policy that still has rollouts in flight.

Versioned adapter names exist to stop cache contamination. vLLM keys its prefix cache by adapter name, so publishing every update under one name means KV blocks computed under the previous weights still match after a swap and the prefill is not redone. A rollout could take its prefix from one policy version and its decode from the next, with no way for the trainer to notice except the ratio metric drifting away from 1.

The proxy exists for two reasons. Exposed Job ports require an authorization header carrying a Hugging Face token on every request, and that header has to be added somewhere. Beyond that, more than one replica means adapter loads must fan out to all of them. In vLLM's data-parallel mode a call to /v1/load_lora_adapter reaches only the rank that answers it, which is why TRL refuses adapter-only sync there, and on Jobs each replica is its own machine so data parallelism moves one level up. From TRL's point of view the proxy is a single vLLM server with data_parallel_size=1.

The router returned 84.5 percent affinity hits across 64,728 rollouts

The proxy's second function is routing for KV cache reuse. GRPO sends G requests sharing one prompt, and G is 8 in this run. If all eight land on the same replica, the first computes the prefill and the other seven reuse it, whereas round-robin would send half to a replica without the prefix and waste that GPU compute.

The router cuts prompts into 16-token blocks exactly as vLLM does and chains the block hashes. The hash of block 3 identifies blocks 1, 2, and 3 together, which mirrors causal attention. The chain is seeded with the adapter name, because a prefix cached for policy v3 is useless for v4.

The awkward part is the common prefix. All 1,460 problems in this run begin with the same 23-token chat template, and that block lands in every replica's cache within seconds. A plain longest-prefix match would make every new prompt look like a hit on the first replica. The router filters it by fan-out instead: a block with several different successors counts as common and is ignored, while a block that always leads to the same successor belongs to a particular prompt.

The placement rule has three branches. If a replica holds specific blocks and sits no more than 8 requests ahead of the least-loaded replica, the request goes there and counts as an affinity hit. If it is further ahead, the router gives up the cache and sends the request to the least-loaded replica as a spill. If no replica holds specific blocks, the prompt is new and goes to the least-loaded replica as unmatched. At the end of the run the counters read 54,712 affinity, 9,196 unmatched, and 820 spills across 64,728 rollouts, or 84.5 percent, 14.2 percent, and 1.3 percent. Since at least one of the eight requests per prompt must be cold, the theoretical floor for unmatched is 12.5 percent.

Adapter broadcast is treated as all-or-nothing. Each replica has its own bucket mount so they do not necessarily see a new adapter at the same moment, and a "no adapter found" error triggers a retry on that replica alone. Any other error unloads the adapter from the replicas that accepted it, so a policy name never exists on only part of the fleet. All 252 adapter loads succeeded over the run, which is 126 syncs times 2 replicas.

The real lesson across five runs is that the bottleneck never stayed in one place

The interpretation from here is ASAP's. This post is not the familiar story of stacking five optimizations for a 3.9x win. Each time, the new bottleneck appeared on the side that had not been touched, and the instrumentation surfaced that immediately.

In run 1 a step took 22.9 seconds, of which forward and backward accounted for 21.9 seconds, or 96 percent. Trainer model FLU was 3.9 percent, rollout wait was 0.02 seconds, and the queue sat pinned at 476 of 512. The second replica was effectively idle. The cause was the reference recipe's per_device_train_batch_size=1, and processing one roughly 1.2k-token sequence per rank 64 times per step is entirely latency-bound on an H200.

Run 2 changed the batch shape rather than the batch size. Keeping 128 completions per optimizer step, the team set token_budget to 16384 and gradient accumulation to 6 so that many samples pack densely into one padding-free row. Samples per row rose from 1.0 to about 12.7, microbatches fell from 64 to 6, forward and backward dropped from 21.9 seconds to 5.6, and model FLU reached 19 percent. The most revealing number is on the generation side: throughput went from 4.6k tokens per second to 25k with nothing changed on vLLM, because the queue was no longer permanently full and the replicas could finally run.

Run 3 turned off one default. AsyncGRPOConfig defaults gradient_checkpointing to True, so every microbatch recomputed its forward during the backward, and the giveaway was a 16k-token row using only 25 GB on a 141 GB H200. Disabling it brought forward and backward to 4.6 seconds, almost exactly one forward less, with model FLU at 23 percent. The queue then fell to 71 and rollout wait rose to 0.6 seconds. The bottleneck had crossed over to generation.

Run 4 exposed a client-side constant, not a limit in vLLM

Run 4 added a third replica and generation moved only from 25k to 26k tokens per second. The third GPU did almost nothing. The reason sat in rollout/inflight, which read 128 in both runs, with the proxy splitting those requests 44, 43, and 41. max_inflight_tasks limits concurrency for the whole rollout worker rather than per replica.

This is the most portable lesson in the post. A 1.5B model on an H200 processes 43 and 130 concurrent sequences at almost the same cost per token, so splitting 128 requests over three GPUs yields nearly the same throughput as splitting them over two. When added hardware does not raise performance, the cause is sometimes a constant in one's own client code, and such constants are usually set conservatively early and never revisited. The team wrote that they chose it that way because they did not know how hundreds of long HTTPS requests would behave through the public Jobs proxy, by which point 130,000 rollout completions had crossed it without a single transport error.

Run 5 changed max_inflight_tasks to 384 and queue_maxsize to 768, and nothing else. With 128 requests per replica, the queue filled to around 690 of 768 and rollout wait fell to 0.03 seconds. Training became the bottleneck again and median step time settled at 4.8 seconds. Over 500 steps, run 1's 3 hours 27 minutes became 53 minutes, model FLU went from 3.9 percent to 23.5 percent, and samples trained rose from 64,000 to 84,078, an increase of 31 percent.

What this record verifies and what it does not are different lists

The most firmly verified item in the Hugging Face run is correctness, because the ratio metric held at 1.000 for every step and across all 126 syncs, meaning the policy vLLM served always matched the one the trainer used to score the rollout. That is direct evidence that file-based synchronization did not silently drift. Mean staleness stayed at 1.5 versions in run 1 and 2.0 in run 5, both under max_staleness of 4.

The dataset choice is deliberate in the same spirit. The team used sail/Sanity-Test-R1D-1.5B, built by generating 40 answers per MATH problem with DeepSeek-R1-Distill-Qwen-1.5B and keeping the 1,460 questions with a success rate between 20 percent and 80 percent. Problems that are neither solved nor hopeless move the learning curve within a few dozen steps, so a silent failure such as one replica serving the base model under an adapter name shows up in the curve. The dataset functions as the test instrument for an infrastructure experiment.

What the record does not establish is equally clear. Reward rose from 0.145 over the first 20 steps to 0.438 over the last 20, but that is a sanity check on a 1.5B math model, not a performance claim. Run 5 ended at 0.416, slightly below run 1, and the post frames this as drawing the same reward curve in less time. It should not be read as evidence of a better model.

Version pinning remains a standing condition. vLLM is pinned to v0.27.1 and the post says to treat that version as part of the recipe. LoRA support arrived through TRL PR #7017 and ships in TRL v1.14, so the path does not exist in earlier releases. Configurations vLLM cannot serve directly, such as DoRA or modules_to_save, fall back to merged-weight sync, and at that moment the few-megabyte transfer this whole design rests on disappears.

For engineering teams the transferable idea is routing around the node constraint

The value of this setup outside Hugging Face lies in constraint avoidance more than cost. Many teams face a shortage of permission to bind several nodes into one training Job rather than a shortage of GPUs. When a shared cluster's allocation policy or a cloud service's job model forbids collective communication across nodes, the standard response has been to shrink the run until it fits inside one node.

The alternative in this post is that shrinking what must be synchronized removes the communication requirement itself. A few-megabyte adapter travels over object storage mounted as a filesystem, and at that point neither NCCL nor a shared disk is needed. The pattern generalizes to stitching loosely connected resources into one training pipeline, including setups that span an on-premises cluster and a cloud provider at once.

The second transferable item is the instrumentation habit. All five runs were diagnosed from four groups of metrics: step time against forward-backward time, rollout wait, queue occupancy against its maximum, and backpressure. A full queue with zero rollout wait and high backpressure means the trainer is too slow, while an empty queue with rising rollout wait and no backpressure means generation is too slow. The post states that the dashboard answered the question within the first ten minutes in each case. Any team running an asynchronous pipeline should wire up those two metric pairs before tuning anything.

The last item is an honest reading of scale. This is a 1.5B model on three H200s with a 1,460-problem dataset, and the post says directly that the Python asyncio proxy avoided becoming a bottleneck because at most 128 non-streaming JSON requests were ever in flight. It adds that a more refined router handling more traffic would need to be written in a faster language. This is a reproducible experiment log, not a certification of a production training stack.

Source: Amine Dirhoussi, Quentin Gallouédec, Kashif Rasul, Sergio Paniego, "Async GRPO with LoRA across HF Jobs: a bucket, a proxy, and no NCCL" (Hugging Face Blog, September 10, 2026) · reproduction code at github.com/AmineDiro/hfjobs-lora-buckets