Researchers at UIUC and Harvard proposed a third pretraining axis beyond parameters and data in "Explorative Modeling," posted to arXiv on July 29, 2026, by factoring the training loop instead of the generation procedure. The method explores K candidate matches between model generations and data at each training step and trains on the closest one, and the authors report that exploration improves FLOP efficiency by 4.1 times, sample efficiency by 6.2 times, and parameter efficiency by 47 percent. On ImageNet 256x256 image generation it reaches an FID of 1.43 without guidance. ASAP works from arXiv 2607.27372 directly to lay out the paper's claims and its limits.

The premise: generative modeling is the holdout that is still not trained end to end

The starting point of arXiv 2607.27372 is the historical lesson AlexNet delivered to deep learning, which the authors treat as the field's default and generative modeling as the exception. AlexNet kicked off the revolution when end-to-end training beat hand-designed layer-wise training, and the pattern carried into image classification, object detection, and image segmentation, with hand-built pipelines broadly replaced by end-to-end networks. The authors argue generative modeling has remained the exception. Autoregressive and diffusion models are trained to predict a single step but used at inference as recurrent networks over hundreds to thousands of predicted tokens or denoising steps, so per-step errors feed into the next step, drifting inputs off the training distribution and compounding.

The paper's explanation is multimodality. Generative modeling is fundamentally about handling probability distributions with many modes, and every scalable approach today handles this the same way, by factoring the generation procedure into smaller steps so that each step's target is nearly unimodal and a reconstruction loss no longer blurs distinct modes into their average. That factorization is exactly what prevents end-to-end generation.

So the paper inverts the question. A generative model has only two processes to decompose, how it generates and how it trains, and with generation-side factoring ruled out, the training loop is what remains. At each training step the model explores K possible matches between what it generates and the data, and trains on the closest. Because this happens entirely during training, Explorative Models capture multimodal distributions while enabling end-to-end generation.

Where the 4.1x, 6.2x, and 47 percent figures come from

The efficiency claims run along three lines: exploration improves FLOP efficiency by 4.1 times, sample efficiency by 6.2 times, and parameter efficiency by 47 percent. These come from adding exploration to the Representation Autoencoder (RAE) image-generation recipe, which was the state-of-the-art ImageNet 256x256 recipe as of three months before this work's release and whose main change was moving the representation space from SD-VAE to a representation autoencoder.

The convergence comparison shows the gains compound. XRAE converges 6.2 times faster than RAE, and RAE itself converges 47 times faster than SiT, which puts XRAE at almost 300 times faster to converge than the standard SiT recipe.

Absolute numbers accompany the ratios. Training XL models for up to 2.2x10^21 FLOPs, an XDiT model with XM-2 reaches 1.43 FID without guidance, below the 1.55 of the RAE baseline under the same condition, and improves FDr6 from 4.42 to 3.91. Under guidance the gFID moves slightly the wrong way, from 1.16 to 1.19, which the authors attribute to gFID being highly saturated and often misrepresenting sample quality, the reason they rely primarily on FDr6.

Why the paper claims the gains grow with scale

The most consequential observation here is a slope rather than a level. Gains from exploration climb from 7 percent to 36 percent as data scales and from 13 percent to 23 percent as models grow, and the efficiency gains more than double at 3 times the compute.

The authors explain this through three capacities. Parameters restrict what a model can represent, data restricts what it can learn, and generative expressivity restricts what it can generate. Conventional scaling relieves the first two, but generative expressivity is set by the training objective itself, so it stays fixed no matter how large models or datasets grow. At small scale the fixed value is not the binding constraint because parameters and data bind first, but as those scale up, generative expressivity moves into the bottleneck position. Exploration raises that expressivity directly, so its benefit should grow with scale, and the observed doubling of efficiency gains when moving from the SiT setting to the 3x-compute RAE setting is offered as evidence.

What makes this framing useful is that it is falsifiable. If generative expressivity really is the binding constraint, gains from exploration should keep growing with scale; if the diagnosis is wrong, they should bend at some size. The evidence in the paper covers XL models up to 2.2x10^21 FLOPs, and whether the slope holds at frontier scale is not yet established.

The trade: paying in training compute instead of inference compute

The terms of the trade are explicit. Exploration happens entirely during training, so inference is unchanged, and the added cost shows up as more expensive training. The authors answer this directly in the paper's FAQ, adding that for Reverse XM the added training cost is often negligible.

The end-to-end experiments make the trade concrete. On robotics control tasks, Explorative Policy rivals Diffusion Policy with a single network forward pass instead of 100, and the Explorative World Model matches Diffuser using 16 to 256 times fewer function evaluations. The paper reads this gap as coming directly from what each approach factors: diffusion pays for its generative expressivity with generation steps at inference, while end-to-end XMs pay for it with exploration during training, keeping inference at a single forward pass.

Industrially this points against the prevailing direction. With inference-time compute established as a main path to better performance, this paper argues that pulling the same expressivity forward into training lets inference finish in one forward pass. For an organization that trains once and serves many times, the arithmetic favors that trade. The authors' note that they obtained these results with barely any hyperparameter tuning, keeping each baseline's architecture and occasionally adding a recurrent block, cuts both ways: untuned numbers show the method's raw strength, but the baselines were tuned no harder, so the gap in a well-tuned production pipeline may differ.

Where the numbers need a careful reading

First, gains are not uniform across domains. The paper reports that exploration monotonically improves performance across images, video, and language, but its limitations section states plainly that autoregressive language models were the hardest to improve, likely because injecting a latent into them is less natural and because they are less bottlenecked by generative expressivity than other models. What the authors obtained there is initial, modest evidence of improved data efficiency. Carrying the 4.1x and 6.2x figures over to large-scale language model pretraining would be a misread; those numbers come from continuous-domain image generation recipes.

Second, the basis for comparison shifts. Exploration changes the training objective, so losses are no longer directly comparable across exploration levels. The authors state this makes distributional metrics like FID and FVD and downstream metrics like accuracy more important. Comparing training-loss curves, a routine habit, does not work under this method.

Third, guidance is unresolved. Existing guidance techniques were designed without exploration in mind and transferred unevenly: autoguidance worked decently, while classifier-free guidance helped less than it does for base models, even though the stronger unguided FID indicates XMs capture the underlying density better. Because exploration supplies a signal base models lack, namely K candidates and a notion of which was best, the authors name guidance designed specifically for XMs as one of the most important open problems.

Limitations and what remains open

The sharpest constraint on fully end-to-end XMs is highly multimodal data, because Forward XM needs K to grow with the number of modes, which is currently too expensive for distributions with very many modes such as image generation. The authors write that while the world remains compute constrained, Reverse XM is the natural solution, since a single model generation can search arbitrarily many training datapoints.

Reverse XM carries its own cost. Being mode-seeking, it requires an entropy term or coverage constraint to avoid collapse, and a simple coverage constraint served that role when training the paper's Reverse XM language models. On whether training on the model's own generations causes collapse, the answer is that training targets are always real data and the model's generations only decide which datapoint each noise input gets paired with; in Forward XM every datapoint still receives a training signal, so no part of the data distribution can be dropped.

The applicability condition is also clear. Exploration helps whenever generation is conditioned on a latent variable the model can search over, such as the input noise in a diffusion model. Continuous generative models benefited most easily because they already condition on noise, while masked diffusion language models benefited once given a learned latent variable embedding for exploration. On how to choose K, the authors say the right amount is problem dependent, since some distributions have many more modes to capture, and that it deserves the same study that scaling parameters and data together has received.

Code is published at github.com/alexiglad/XM. Whether the phrase "third axis" holds up depends on independent reproduction and validation at larger scale, and whether meaningful gains appear for autoregressive language models is the fork that decides how wide this method's applicability turns out to be.

Source: ASAP analysis based on arXiv 2607.27372, "Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation" (Alexi Gladstone, Heng Ji, Yilun Du, posted July 29, 2026)