World models that skip human minds predict the wrong action: F1 climbs from 63.3 to 87.9 on Menti-Bench

A world model that tracks only the physical scene scores F1 63.3 at predicting what a person will do next, while a pipeline that treats beliefs and intentions as state variables reaches 87.9 on the same task, according to arXiv paper 2607.27201. "Mental World Modeling," by Hao Fei of the University of Oxford and Yiran Zhao of the National University of Singapore, was posted on July 29, 2026, and the Mentis code and the 448-item Menti-Bench evaluation set followed in August. Humans score 98.5 under the identical protocol. ASAP summarizes the result from the paper and the repository as primary sources.

Perfect physical understanding does not guarantee the right action

World models have so far been machinery for answering a physical question: what is where, and how will it evolve. The problem the paper identifies is that human behavior is not determined by that information alone. If an object was moved while someone was away, that person searches where they last saw it, not where it actually is. If an agreement was quietly broken and the party does not know, they still act on it. The scene can be read perfectly and the action prediction still comes out wrong.

MWM addresses this by redefining what the world state contains. State becomes a single coupled physical-mental value; a partial observation is rendered for the target agent covering only what they can actually see, hear and know; and each candidate action is simulated for how it changes both the scene and the minds inside it. The key move is promoting beliefs and intentions from post-hoc rationales to state variables. A wrong rationale leaves no trace, but a wrong state feeds the next stage and makes the result diverge.

Mentis, the implementation, is a training-free six-stage pipeline: state parsing, target-observation rendering, action decomposition, coupled physical and mental transition, branch evaluation, and decision. Every stage logs its artifact, so a wrong prediction can be localized to a bad state parse, an observation that leaked inaccessible facts, an implausible transition, or a poor evaluation. The final decision is made deterministically outside the LLM, weighting mental consistency, physical plausibility and social appropriateness at 0.45, 0.35 and 0.20.

Climbing the ladder one rung at a time, then removing channels

Evaluation runs over all 448 Menti-Bench records, reported as final-action F1. Eight models fill the world-model role: five from OpenAI (gpt-5.6-sol, gpt-5.5, gpt-5.4, gpt-5.4-mini, gpt-4.1) and three from Anthropic (claude-fable-5, claude-opus-4-8, claude-haiku-4-5). The ladder S0 through S6 is reported as their average.

The choice-only floor, where the story is withheld, sits at 31.3, with all eight models packed between 29.7 and 33.2. A stronger model extracts nothing more from the options alone. Letting models read the story (S1) lifts this by 32.0 points to 63.3. Chain of thought (S2) reaches 74.6, and six-sample self-consistency (S3) reaches 77.9. The modeling rungs begin here: an unformatted world-state note (S4) gives 80.3, typing that state into the physical-mental schema (S5) gives 82.6, and adding observation rendering, branch simulation and value evaluation for full MWM (S6) gives 87.9.

One contrast stands out. The unformatted state note alone (80.3) already beats six-sample self-consistency (77.9). And S3 trails S6 by 10.0 points on average, with the weakest model's S6 (gpt-4.1 at 84.9) exceeding the strongest model's S3 (gpt-5.6-sol at 83.6). Spending more compute does not close this gap.

The ablations confirm three structural claims separately. Removing the mental channel costs 12.1 points on average, removing the physical channel costs 16.5, and predicting the two transitions independently costs 6.4. The ordering, with S6 highest, then decoupled transitions, then mental removal, then physical removal, holds for all eight models. Physical modeling alone is insufficient, mental reasoning degrades without physical grounding, and the joint transition beats handling the channels separately.

Weak models need the structure more than strong ones do

The most counterintuitive result is that the benefit of explicit structure shrinks as the base model gets stronger. The S6 minus S1 gain is 21.1 points on gpt-5.6-sol against 28.0 on gpt-4.1, and 22.1 on claude-fable-5 against 25.8 on claude-haiku-4-5. Both families move the same direction, so this is not one vendor's quirk.

How to read that number is where interpretations diverge. The optimistic reading is that stronger models make explicit structure redundant and scale eventually absorbs the problem. The paper does not read it that way, and the data supports its caution: the gain shrinks without disappearing, and even the strongest model stalls at 69.6 under direct answering. The more consequential practical implication runs the other way. If weaker models gain more from structure, then engineering the pipeline can partly substitute for upgrading to a more expensive model API, which opens room to recompute the cost side.

Where the gains concentrate is equally clear. Interpersonal decisions are the weakest category under direct answering (66.5) and the strongest under full MWM (92.9), producing the largest gain at 26.4 points. Object and resource scenes gain only 14.0, because their direct-answering score already sits at 74.0; scenes like these are largely solvable from commonsense affordances. The spread between categories narrows from 7.5 points under S1 to 4.9 under S6, with all four categories at or above 88.0. The benefit is largest exactly where hidden mental variables govern the decision.

The bottleneck is transition simulation, not state parsing

The most practical section is the set of four oracle interventions, substituting gold state, observation, action or transition into the gpt-5.6-sol pipeline one stage at a time. Swapping a gold annotation into one stage while leaving the rest predictive reveals how much error that stage was contributing. On gpt-5.6-sol, fully predictive S6 scores 90.7, leaving 7.8 points to the human reference.

Gold state lifts it to 93.5 (+2.8), gold observation to 92.4 (+1.7), and skipping action decomposition to use the option text verbatim to 91.4 (+0.7). The largest single-oracle gain comes from gold transitions at 94.2, or +3.5. Imagining the next state from a given state is harder than reading the current state correctly.

That ordering sets the priority for follow-up work. If prompted simulation of successor states is what binds performance, learning transitions from data is the natural next step. The difficulty is that mental successor states are only partly observable. The paper proposes indirect supervision through counterfactual pairs, human rationales, later behavior and consistency over time, while explicitly resisting collapsing all of it into a single end-to-end predictor, since keeping physical and mental factors separable is worth 6.4 points from the coupling alone.

What conversational product teams should take from this

The failure this paper quantifies, worth 26.4 points on interpersonal scenes, is the one that breaks multi-turn support, commerce and tutoring products. Teams building them generally handle summarization and intent classification well, and what breaks is the next layer. Unless the system holds, as separate state, what this user currently misunderstands and what they do not yet know, it repeats guidance already given or speaks from facts the user has never seen. That failure is precisely what this paper quantifies.

Two prescriptions transfer directly. First, give the dialogue state explicit fields for the other party's beliefs and observational access, not just facts. That a free-text state note alone beat self-consistency sampling speaks to the cost-effectiveness of this step. Second, when selecting among candidate responses, simulate how each changes the other party's mental state. The caveat is cost: Mentis spends roughly 12 LLM calls and 48,000 tokens per sample. That is not a structure to drop into real-time conversation, so latency-tolerant paths are the realistic starting point.

What 448 items can and cannot settle

Menti-Bench comprises 320 text records, 100 image records and 28 sounding-video records. Twenty-eight videos is a small sample for statistical claims, and the paper states that absolute deltas there are treated as directional only. Wrong options are hardened so each violates at least one constraint inferable from the scene, whether a character's belief or perceptual access, an object's state, an agreement, a norm or timing, and intermediate gold annotations are withheld so the benchmark cannot be shortcut with oracle information. Code is MIT-licensed and the data is CC BY-NC 4.0.

Some limits on the numbers themselves. Every run shares a single fixed operating point of medium reasoning effort with batched comparative-rank scoring, and whether the ordering survives at other operating points is outside this experiment. Oracle interventions ran on gpt-5.6-sol only, so it is unverified that the bottleneck sits in the same place for every model. Mentis is a training-free baseline over an OpenAI-compatible endpoint, which leaves open whether these gaps persist in trained systems.

The paper's own closing concern is the most interesting one. Full MWM with gold intermediates reaches 97.0, humans reach 98.5, and the strongest predictive configuration already sits at 90.7. As scores compress toward the ceiling, an outcome number stops distinguishing systems. A model that lands on the right answer from an unfaithful state collapses as soon as inputs drift, yet looks identical to a sound one in an accuracy table. That is why the process-level gold and judge suite exist, and why the authors argue for making process faithfulness a training objective rather than a diagnostic. For models that simulate minds, counting why the answer was right has to come before counting whether it was.

Source: ASAP summary based on Hao Fei and Yiran Zhao, "Mental World Modeling" (arXiv:2607.27201, July 29, 2026) and the GitHub repository mental-world/Mentis.