Researchers from KAIST and Naver AI Lab propose Stable-GFlowNet (S-GFN), a framework that secures attack diversity and attack success at the same time in LLM red-teaming. Across 1,024 generated prompts, the count of semantically distinct attack types rises from 17.67 for the existing GFlowNet method to 134.00, roughly seven times more, while the attack success rate holds at 92.55%. The paper is arXiv:2605.00553, first posted on May 1, 2026, and according to KAIST it was accepted as an ICML 2026 Spotlight. ASAP works from the paper itself as the primary source to explain what this method actually solves.

The attacks succeed, but they are always the same attack

Automated red-teaming fails through repetition rather than through low success rates. In Table 1 of the paper, PPO reaches a 91.70% attack success rate against the target victim model while producing only 3.00 semantically distinct attacks. By success rate alone it ranks near the top; in practice it repeated essentially one attack 1,024 times. Adding a curiosity bonus barely helps, yielding 4.00 unique attacks at a 36.75% success rate.

The reason this matters becomes visible from the defense side. The purpose of red-teaming is not to prove one vulnerability but to enumerate as wide a set of trainable vulnerabilities as possible. A 91.70% obtained by repeating one attack is a number that a single round of safety fine-tuning erases, and it cannot serve as evidence that the model became safe. This is the familiar disappointment for organizations that adopt automated red-teaming tools: the reported success rate is high, and the defense built from those results barely holds against real user attacks.

That is why GFlowNet (GFN), which targets distribution matching, drew attention as an alternative. Existing GFN is notorious for training instability and mode collapse, and in Table 1 it produces 17.67 unique attacks at a 93.75% success rate, more diverse than PPO but still narrow.

Removing the denominator: CTB, NGP and MKS

The starting point of S-GFN is the elimination of the partition function Z estimate from the GFN training objective, and three components are built on top of that change. Contrastive Trajectory Balance (CTB), the first component, replaces absolute balance on a single trajectory with a pairwise comparison between trajectories. Because Z is never estimated, the instability originating in that estimate disappears, and the paper offers a proof that CTB preserves the optimal policy of GFN.

Noise Gradient Pruning (NGP), the second component, addresses reward noise. In red-teaming both the autoregressive generation of the victim LLM and the toxicity classifier introduce variance, so the reward signal for the same prompt fluctuates between runs. NGP is a conditional variance-reduction technique that prunes the gradients this noise produces.

The third component, the Min-K Fluency Stabilizer (MKS), targets a flaw in the reward model itself. Toxicity classifiers assign arbitrary middling scores around 0.2 to 0.3 to out-of-distribution gibberish, and the attacker model turns this into a reward-hacking channel, collapsing into a local optimum that emits meaningless strings. The conventional remedy, KL-divergence regularization, prevents that failure but confines samples inside the reference model's distribution and loses diversity. MKS instead averages the probabilities of the k least likely tokens rather than summing log-probabilities over the whole sentence. The paper argues that the log-probability sum is sensitive to sequence length and penalizes rare tokens such as proper nouns, which narrows the search space.

What was measured, and how

The evaluation setup is built around control rather than scale. The attacker LLM is Qwen2.5-1.5B supervised fine-tuned on Safety-Dataset and AdvBench, the victim LLM is Qwen2.5-1.5B-Instruct, and the toxicity classifier is Meta-Llama-Guard-3-8B. Attack success rate (ASR) is the proportion of the 1,024 generated prompts that elicit a victim response with a toxicity score above 0.5, averaged over five sampled responses per prompt. Unique Attacks (UA) counts distinct clusters from greedy clustering at a threshold of 0.7 over all-MiniLM-L6-v2 sentence embeddings.

Those definitions are what make the table readable. UA is sensitive to the clustering threshold, so its relative comparison under fixed conditions carries the meaning rather than its absolute value, while ASR measures attack effectiveness independently of that threshold. Reporting both is what lets the authors claim the S-GFN result is not an artifact of finely split clusters. S-GFN delivers 134.00 UA and 92.55% ASR together, with standard deviations of 12.77 and 2.87 over three trials.

The baselines are ICL, SFT, DPO, PPO, PPO+Curiosity, GFN (TB), Jailbreak-R1 and Rainbow Teaming. The paper notes that Jailbreak-R1 runs the official 8B checkpoint with chain-of-thought enabled at inference only, so it is treated as a strong baseline rather than a like-for-like comparison.

The defense experiment is the more interesting half

The most operationally relevant result in the Stable-GFlowNet paper is defensive rather than offensive, since a model fine-tuned on S-GFN attacks holds the GFN attack to a 0.03% success rate. The authors built a safety dataset from the attacks each method discovered, fine-tuned the victim model on it, and then cross-attacked the four resulting defense models. The outcome is asymmetric.

A model defended with S-GFN-discovered attacks blocked the GFN attack at 0.03% and allowed only 0.75% even against S-GFN's own attacks. In the reverse direction, S-GFN attacks broke most other defenses: 83.24% success with 110.00 unique attacks against the Rainbow Teaming defense, 56.25% with 87.67 against the Jailbreak-R1 defense, and 22.53% with 43.33 against the GFN defense.

The consistency of that direction is the point. A defense trained on a broadly discovered attack set also blocks attacks found by other methods, while a defense trained on a narrow set collapses as soon as the angle shifts slightly. Rainbow Teaming's particular fragility has an explanation: it diversifies attacks inside a predefined matrix of styles and topics, so its defense does not generalize beyond that matrix. For anyone commissioning or running a safety evaluation, the implication is direct. Score a red-teaming tool on success rate alone and the narrow tool looks better, and the defense built from it is the one that fails first in production.

What the transfer results actually show

The transfer experiments show both the range over which S-GFN's advantage holds and how low the absolute numbers are. Attacking unseen victims without retraining, S-GFN records 34.67 unique attacks at 15.04% on Gemma3-4B-Instruct, 52.00 at 15.01% on Llama3.2-3B-Instruct, 37.33 at 0.18% on Qwen3-4B-Instruct and 90.00 at 0.51% on gpt-oss-20B, leading every baseline on all four.

Relatively, that is a clear win. The absolute figures deserve caution. A rate of 0.18% on Qwen3-4B-Instruct and 0.51% on gpt-oss-20B correspond to roughly two and five successes out of 1,024 attempts. An attack that achieves 92.55% against a 1.5B victim therefore falls below 1% once it moves to a current-generation model. The right reading is that recent safety alignment has strengthened substantially rather than that the method is weak. It does mean that carrying a "92% red-team success rate" headline over to frontier models is not supported.

How far the method generalizes

The generality claim of Stable-GFlowNet is supported by two experiments outside red-teaming entirely, in molecular generation and in a noisy hypergrid. The authors compare S-GFN on molecular generation over QM9, building length-10 molecules from 10 fragments and reporting average QED, and on a noisy hypergrid setting reporting log Jensen-Shannon divergence. The framing is a GFlowNet training-stabilization method rather than a red-teaming-specific trick.

The ablation separates the contribution of each component. Under the MKS reward setting, GFN-TB yields 67 unique attacks at 85.8% ASR, switching to CTB yields 108 at 82.9%, and adding NGP yields 121 at 92.2%. CTB buys diversity, and NGP raises diversity and success rate together. Comparing reward settings, using the raw reward R(y) with no constraint collapses both TB and CTB to 0 unique attacks, log-probability constraints give 65 and 78, and MKS gives 67 and 108. Reward design moves the outcome as much as the learning algorithm does.

One discrepancy is worth flagging: the ablation's 121 differs from the 134.00 of the main table. The main result carries a standard deviation of 12.77, so the two sit within one deviation, but the paper does not connect them explicitly. When citing the full three-component configuration, the three-trial mean of 134.00 ± 12.77 is the accurate figure.

The open questions for practitioners

Two levels of the Stable-GFlowNet result are relevant to Korean organizations, and the first of them is authorship. The paper lists KAIST and Naver AI Lab, with one corresponding author at each, making it a case of a Korean university and a Korean corporate lab jointly producing a top-tier venue result in LLM safety, and the code is public at github.com/kmc0207/Stable-GFN for reproduction and extension.

The second is the cost of adoption. The experiments ran with a 1.5B attacker, a 1.5B victim and an 8B toxicity classifier, a configuration far cheaper than API-based red-teaming against frontier models. But as the transfer results show, attacks found against a small victim do not carry over intact to a large one. Deploying this method in practice presupposes retraining both the attacker and the victim against the actual production model.

The remaining question is multilingual coverage. The experiments use English datasets and an English toxicity classifier, and no results are reported for Korean attack prompts. Applying this framework to a Korean-language service requires attaching a Korean toxicity classifier as the reward model and re-measuring that classifier's noise characteristics first. Given that reward-model noise is precisely the problem MKS was designed to solve, that re-measurement is the step on which the port succeeds or fails.

Source: ASAP analysis based on Minchan Kwon et al., "Stable-GFlowNet: Toward Diverse and Robust LLM Red-Teaming via Contrastive Trajectory Balance" (arXiv:2605.00553, KAIST and Naver AI Lab)