ComPO Aligns LLMs Without Differentiating a Preference Loss, Updating Just 0.02% of Parameters

ComPO is a zeroth-order alignment method that updates a language model from one-bit comparison signals instead of the gradient of a differentiable preference loss, released on September 16, 2026 as arXiv 2609.19144 by researchers at UC Berkeley, NYU, Alibaba's DAMO Academy, and Columbia University. The update touches roughly 1% of output-layer entries, about 1.5 million parameters for Mistral-7B, or about 0.02% of the full 7B model. On Llama-3-8B-Instruct the AlpacaEval 2 length-controlled win rate moves from 32.59% under DPO to 35.79%, and on a released Gemma-2-9B-it SimPO checkpoint from 60.36% to 62.42%. ASAP reads this paper along one axis: what you gain and what you give up when you abandon the gradient.

Dropping the gradient for a single yes-or-no bit

ComPO is an alignment method that removes gradient computation from the training loop entirely, replacing the gradient of a differentiable preference loss with aggregated one-bit answers from a comparison oracle. Comparison-based Preference Optimization perturbs the current policy along random directions and asks only whether each perturbation raises the log-likelihood of the preferred response while lowering that of the dispreferred one. Each answer is a single bit, and the bits collected across many perturbations are aggregated into a normalized update direction. As the abstract states, the method extracts directional information from low-margin pairs without directly optimizing a differentiable preference loss on them.

The design works because the test requires only two log-likelihoods. Obtaining the gradient of a loss function requires backpropagation through the whole model, while comparing two log-likelihoods requires only forward computation. The paper states that the comparison-based update avoids full-model backpropagation and does not store all perturbation vectors.

The update scope is narrow as well. Outside the multi-layer ablation, every parameter beyond the output layer stays frozen, and a threshold retains only the output-layer entries with sufficiently large magnitude. For Mistral-7B the output layer holds 0.13B parameters, and retaining 1.18% of its entries corresponds to roughly 1.5 million updated parameters, about 0.02% of the full 7B model. Sweeping that threshold, the strongest results appear when approximately 1% to 6% of entries are retained; keeping many small entries or filtering almost all of them both degrade performance.

The low-margin pairs everyone else discards are the raw material

The most consequential practical decision in this paper is splitting the data in two. The authors group pairs whose preferred and dispreferred log-likelihoods differ by less than a margin threshold under the reference model into a noisy set, leaving the rest as a clean set. The paper is explicit that "noisy" names this low-margin subset and does not presume the preference labels are wrong. Training then runs in two stages: a direct alignment method such as DPO or SimPO is applied to the clean set first, and ComPO is applied to the noisy set on top of the resulting checkpoint.

The target of that split is likelihood displacement. When the two responses in a pair look similar to the model, direct alignment methods that push the dispreferred response down can drag the preferred response's likelihood down with it. Discarding such pairs shrinks the dataset; training on them directly steers the model wrong. ComPO takes a third route, treating those pairs as comparison signals about a latent alignment objective rather than as samples for a fixed loss.

Table 2 is the direct evidence. On Llama-3-Instruct-8B the preferred and dispreferred log-likelihoods start at (-46.761, -47.410), and across three independent trials at a step-size setting of 1 the preferred value rises to -46.728, -46.743, and -46.753 while the dispreferred value falls to -47.520, -47.525, and -47.517. On Gemma-2-9B-it the pair starts at (-133.122, -134.557), with the preferred value rising to -133.059 and the dispreferred falling to -134.565. The signature pattern of likelihood displacement, both values sinking together, does not appear.

Scaling the noisy set confirms its role as raw material. Increasing the number of noisy pairs from 100 to 300 raised mean performance on both AlpacaEval 2 metrics and on Arena-Hard. Data that existing pipelines filter out as low quality becomes, in this method, a resource that pays more the more of it you have.

Tables 1 and 3, and the one cell that moved backward

The measured gains are most consistent on the AlpacaEval 2 length-controlled win rate. For Llama-3-8B-Instruct that figure reads 32.59% for DPO, 32.92% for DPO on the clean set alone, and 35.79% once ComPO is added. The raw win rate moves from 31.99% to 35.03% and the MT-Bench average from 7.93 to 8.05. Mistral-7B-Base rises from 9.71% to 11.66%, and Llama-3-8B-Base from 4.14% to 5.39%.

Adding the method to an already-released checkpoint behaves the same way. Table 3 applies ComPO to public SimPO checkpoints with no other change: Mistral-7B-Instruct goes from 40.22% to 42.27% length-controlled, Llama-3-8B-Instruct from 48.71% to 49.53%, and Gemma-2-9B-it from 60.36% to 62.42%. The paper describes this as augmenting other direct alignment methods without changing their original training objective.

Two caveats belong with those numbers. First, the size of the gain depends on the benchmark. In the same Table 3, Arena-Hard win rates move only from 20.8% to 22.0% and from 36.3% to 37.3%, and on Gemma-2-9B-it the figure stays at 61.1%. A two-point gain on AlpacaEval 2 corresponds to about one point, or nothing at all, on Arena-Hard.

Second, not every cell improved. On Mistral-7B-Instruct the length-controlled win rate rose from 24.14% to 26.17% while the Arena-Hard win rate fell from 14.4% to 10.5%, a drop of 3.9 points, even as the MT-Bench average jumped from 5.86 to 7.69. The metrics point in different directions in that configuration, and the warning against reading a single benchmark gain as a general improvement sits inside the paper's own table.

Online ComPO brakes with a KL radius

Online ComPO is the extension that keeps the offline comparison mechanism intact and adds unlabeled generations from the current policy, using them to estimate reverse KL divergence against a reference policy. Offline ComPO draws comparison signals only from preference pairs, and the online variant leaves that source unchanged while adding a second measurement over the policy's own outputs. A candidate update is formed, and it is accepted when it stays within a fixed KL radius and damped when it does not.

Table 10 stacks the gains step by step. On Qwen3-4B-Base the AlpacaEval 2 length-controlled win rate reads 15.28% for DPO, 16.20% with ComPO added, 17.43% with reverse-KL control, and 18.57% with resampling, while Arena-Hard climbs from 29.3% to 32.6%. Gemma-3-4B-it moves from 38.30% to 42.55% and its Arena-Hard score from 56.9% to 63.7%. Llama-3.2-3B-Instruct shows the smallest gain, from 11.72% to 13.05%.

The Arena-Hard response deserves attention. Arena-Hard barely moved in the offline configurations, yet after reverse-KL control is added Gemma-3-4B-it jumps from 57.7% to 63.3%, a gain of 5.6 points. The brake that keeps the policy from drifting far from its reference does more work on the harder task set. The paper presents the length-normalized damping rule this online variant uses as a practical prescription in Algorithm 4, and does not place it inside the scope of the convergence guarantee established earlier.

How to read the 23GB figure

The resource numbers are striking, and they are not a like-for-like comparison. The paper reports a peak of approximately 23GB per A40 GPU for Llama-3-8B ComPO, against reported peaks of 77GB for DPO and 69GB for SimPO on H100 GPUs. The very next sentence states that because these measurements use different hardware they describe practical resource requirements rather than a controlled head-to-head comparison. Removing backpropagation lowers memory for structural reasons, but the paper does not license reading 77 against 23 as an efficiency multiple.

A more accurate reading is that the cost moved to a different axis. All ComPO runs used 30 NVIDIA A40 GPUs with 46GB each, and 600 perturbations completed in 50 seconds. Table 4 shows that more perturbations buy more performance: the mean AlpacaEval 2 length-controlled win rate rises from 24.72% at 800 perturbations to 25.02% at 1600, 25.91% at 3300, and 26.49% at 5400. The method trades per-device memory for forward-pass count.

That trade determines who should adopt it. An organization that struggles to secure a few high-HBM accelerators but can run many mid-tier GPUs in parallel is on the favorable side of the trade, and an organization limited in device count is not. The paper's remark that ComPO parallelizes naturally is both a statement of an advantage and a statement of a precondition.

What it offers teams building on public checkpoints

The practical reach of this work is the entry cost of an alignment pass. Most teams fine-tuning their own models start from a public checkpoint, and redoing alignment usually means going back through data curation and the full training loop. Table 3 demonstrates a different path: bolting an extra stage onto an already-aligned SimPO checkpoint without touching the original objective, which costs far less to attempt than redesigning training from scratch.

Confining updates to the output layer also helps operationally. When the changed parameters amount to 0.02% of the model, storing the delta separately and rolling it back is straightforward. The flip side is that the method does not rebuild the model's deeper representations. In the multi-layer ablation, perturbing the MLPs in layers 30 and 31 of Mistral-7B-Instruct alongside the output layer improved all three reported metrics, with peak memory rising from 16.3GB to 16.7GB and 600 perturbations taking 60 seconds instead of 50. Widening the scope buys more, but most of the reported results come from the output layer alone.

The binding constraint is data. The raw material here is the subset of preference pairs whose two responses sit close together under the reference model, and identifying them requires one pass of reference-model log-likelihoods over the whole dataset. Recycling pairs that would otherwise be discarded is a real gain, but a team whose preference data numbers in the hundreds has little room to reproduce the benefit observed when noisy pairs grew from 100 to 300.

The gap between the theorem and the implementation

The theoretical and experimental contributions of this paper do not cover the same object. The authors establish a convergence guarantee for the basic offline scheme under smoothness, gradient sparsity, and compatibility between the oracle and a latent objective, and show that for a fixed sparsity level the number of comparison queries depends only logarithmically on the ambient dimension. Yet, as the paper notes, the objective is latent, so its gradient norm is unavailable as a practical stopping criterion, and the implementation approximates the basic estimator to accommodate the scale of LLM fine-tuning.

The gap is wider online. The performance guarantee is established for a basic constrained scheme under local coverage and in-distribution pairwise reward accuracy, while the best numbers in Table 10 come from Algorithm 4 with length-normalized damping and replay. The scheme that carries the guarantee and the scheme that produced the numbers are not the same one, which is a familiar seam where optimization theory meets large-model experiments. What a reader takes from this paper is empirical evidence that comparison signals alone recover a usable direction, not the safety net of a convergence proof.

Scale remains open. The largest model in the experiments is Gemma-2-9B-it, and the online runs sit between 3B and 4B. Retaining 1% of output-layer entries may behave differently as vocabulary size and hidden dimension grow, and whether aggregating thousands of one-bit signals preserves the same signal-to-noise ratio in a much larger output layer is a question this paper does not answer. Table 4 showing continued gains out to 5400 perturbations is good news and an open question at once, because nobody yet knows how far that count must rise for larger models.

Sources: arXiv 2609.19144, "A Zeroth-Order Paradigm for LLM Preference Alignment" (submitted September 16, 2026, by Peter Chen, Xi Chen, Wotao Yin, and Tianyi Lin), drawing on the abstract, Section 4, and Tables 1, 2, 3, 4, and 10. The earlier version by the same authors is arXiv 2505.05465, "ComPO: Preference Alignment via Comparison Oracles" (submitted May 8, 2025).