HumanCLAW measured whether vision-language models can act through a body: the best score was 16.8%

HumanCLAW, released on August 3, 2026 by researchers from Meta, Nanyang Technological University, the University of Washington, Brown University, and Northwestern University, is an evaluation framework that measures whether a vision-language model can operate as an embodied agent. The benchmark comprises 1,218 long-horizon egocentric episodes across 41 indoor scenes in which an agent must find an object, walk to it, and sit on it, and none of the nine state-of-the-art models tested solved it. Gemini-3.1 posted the best result at 16.8%, and the bottleneck the paper identifies is not object recognition but a missing capacity the authors call embodied self-awareness. ASAP works from arXiv 2607.27180 to lay out what this benchmark separated in order to measure it.

Half-physics strips out the body and tests only the decisions

HumanCLAW's defining design choice is the separation of action decisions from motor control. In conventional embodied AI evaluation, a sound decision still fails the task if joint control breaks down, so decision quality and motor quality collapse into a single score and the cause cannot be recovered. The researchers instead have the vision-language model issue only atomic skill commands, which a physics simulator translates into continuous full-body motion under working gravity and collision.

The command vocabulary is fixed at eight entries: walk(x,z,ψ), side_step(x), step_back(z), turn_in_place(θ), climb_upstairs(h,d), walk_downstairs(h,d), sit_in_place(h), and stop. The paper calls this arrangement half-physics, meaning the simulator absorbs balance and motor tracking while collisions and terrain remain fully live.

The task itself runs in three stages. From an egocentric view alone, the agent locates a target object category, approaches within 20 centimeters, and finally sits on it. The 1,218 episodes were generated from 41 HSSD validation houses and split into a sit subset of 597 episodes targeting beds, couches, and toilets, and a navigation subset of 621 episodes targeting chairs, potted plants, and TVs. Each episode is stratified by three axes: geodesic distance to target, turns plus rooms traversed, and average obstacles within one meter of the route.

The scoreboard: finding works, sitting does not

Among the nine models evaluated, Gemini-3.1 completed all three stages most often at 16.8%, and the remaining eight landed in single digits or near zero. Stage-level success rates (FindSR, NavSR, InteractSR) follow.

Model	Find	Navigate	Sit
Gemini-3.1	64.9%	42.4%	16.8%
Gemini-2.5	58.5%	21.6%	3.5%
GPT-5.5	55.1%	13.9%	3.4%
Gemma-4-31B	58.1%	28.7%	11.1%
Qwen3.6-27B	51.0%	20.9%	0.2%
InternVL3.5-38B	46.8%	0.8%	0.0%
Qwen3.6-35B-A3B	44.6%	5.8%	0.0%
Qwen3.5-27B	37.8%	13.5%	0.0%
Claude-4.8	32.6%	8.6%	1.5%

What stands out in this table is not the ranking but the slope of each row. Gemini-3.1 falls from 64.9% to 16.8%, roughly a quarter of where it started, and InternVL3.5-38B finds the target 46.8% of the time yet collapses to 0.8% in navigation. The paper reports no human baseline, so these figures are meaningful only relative to one another.

The decay curve is the real finding

What separates HumanCLAW from prior evaluations is that it leaves behind three stage-level scores instead of one success rate. Had it measured recognition alone, Gemini-3.1's 64.9% would be the headline and the story would end at the familiar observation that frontier models locate indoor objects fairly well. Split into three stages, the same model produces a descending sequence of 64.9%, 42.4%, and 16.8%, and the question that follows is a different one entirely: knowing what to look at and moving your own body in front of it are separate capabilities.

That distinction runs against the direction multimodal models have improved in for several years. Visual understanding has been measured by how accurately a model describes a single image, and along that axis progress has been fast. HumanCLAW asks instead for many frames stitched in temporal order, with a running estimate of where the body currently is and how much distance remains. The first capability is static observation; the second is closer to state tracking.

The value of the half-physics design surfaces here. Because the simulator handled motor control and the score still stopped at 16.8%, poor walking is not what prevents humanoids from doing work. Assume perfect gait stability in hardware and this benchmark's numbers stay where they are. The diagnosis the design yields is that the bottleneck sits in the head that commands the body, not in the body.

Two thirds of failures come from not knowing where the body is

The largest share of failures traces to body-awareness errors rather than visual recognition. Across all models, 68% of the 5,473 episodes in which the target was found went on to fail navigation, and by cause, 30% stopped far from the target through ego-spatial distance hallucination, 20% arrived but never signaled arrival, and 14% jammed against geometry while continuing to command forward motion. Those three sum to about 64%. In the same stage, mid-level reasoning accounted for 17% of failures, visual perception for 9%, and locomotion for 10%.

The interaction stage is skewed further. Of the 726 episodes that navigated successfully, 71% failed to sit, and 81% of those failures were body-awareness errors: 58% attempted to sit into thin air with no contact, 14% sat on the wrong object, and 9% reached the seat without stopping on it. Decision failures, where the sit command was never issued at all, made up 19%.

The finding stage breaks down differently. Mid-level reasoning failures including ineffective exploration were largest at 38%, followed by perception lapses where the target was rendered but unacknowledged at 23%, approaching without viewing the target at 10%, ineffective actions at 12%, and undetected jamming at 9%. Collision analysis places legs and feet at 28% to 45% of collision steps, arms and hands at 20% to 35%, torso at 10% to 24%, and head under 7%.

This distribution points at where improvement has to come from. Visual perception failures at 9% in the navigation stage mean better eyes will not move this score. What is required is a representation that maintains body position and state over time, which is a state-management problem rather than a vision-encoder problem.

Feeding in more images makes performance worse

The most practically valuable result in the ablation study is that adding visual context degraded performance. On a 100-episode mini-validation set, the baseline configuration of ten steps of text history and one image scored 27.0% navigation success, and raising the image count to ten cut that to 13.0%, less than half. The curve rises slightly to 31.0% at two images, then falls to 28.0% at five and 13.0% at ten.

Text history behaves differently. Removing it entirely drops navigation success to 11.0%, confirming that history is necessary, but the score is effectively flat from there: 27.0% at ten steps, 27.0% at twenty, 28.0% at fifty, and 26.0% at one hundred. In the paper's words, text history is necessary but saturates quickly.

The decisive component is the skill verifier. Removing it collapses navigation success from 27.0% to 2.0% and sitting success from 18.9% to 0.0%. Removing the mid-level reasoning module actually raises finding success to 62.0% while driving sitting success to 0.0%.

Taken together, these three results overturn a common assumption. Adding context, particularly pushing in more visual frames, is not the improvement path for embodied tasks. Ten images actively hurting fits an interpretation in which the model confuses recent frames with older ones instead of integrating them temporally, which is the same phenomenon as the 30% distance hallucination seen from a different angle.

How to use these numbers in a robotics purchasing decision

For organizations evaluating humanoids or service robots, the first implication is that the performance question needs rewriting. What matters in a vendor demonstration is not whether the robot recognizes objects but whether it decides on its own that it has arrived and stops. With 30% of navigation failures caused by distance hallucination and 20% by failure to signal arrival, the line between a successful demo and a failed deployment runs through termination judgment, not recognition accuracy.

The second implication concerns architecture. A collapse from 27.0% to 2.0% when the verifier is removed argues for never wiring a language model's judgment directly into execution, and for placing a separate layer that checks each command's feasibility. For repetitive fixed-route work such as warehouse logistics or facility inspection, the design of that verification layer governs real success rates more than the model's autonomous judgment does.

The third implication is timing. A best score of 16.8% means failure more than eight times out of ten, so dropping a current-generation vision-language model into an unsupervised autonomous segment does not work. A configuration where a person sets the target and the robot handles only short autonomous stretches is governed by a different number, the 64.9% finding rate. Success rises as autonomous segments shorten, so deployment is best approached as a segmentation problem.

The conditions that limit how far these results generalize

The first constraint the paper states is the half-physics abstraction itself. Because the simulator handles balance and motor tracking, there is no guarantee that a competent decision maker scores the same once transferred onto physical hardware. The researchers explicitly label this transfer a working hypothesis rather than a verified conclusion.

The second is the missing sensory channel. The interface simulates no tactile feedback, so collisions displace the world but are never felt. The 14% of failures where the agent jammed and kept commanding forward cannot be read apart from that condition, since real robots carry joint torque and contact sensing and would not reproduce that failure at the same rate.

The third is vocabulary size. Because interaction reduces to sitting, the embodied intelligence measured here excludes picking objects up, moving them, or using tools. Decision-level attribution is likewise relative to the eight-skill vocabulary, so a different vocabulary would redistribute the failure taxonomy.

The fourth is the state of the models evaluated. The researchers tested frozen off-the-shelf models with no additional training and state that whether targeted training for body awareness can lift these ceilings remains an open question, one they consider tractable. The fifth is the absence of a human baseline, which means this paper alone cannot say what 16.8% represents relative to a person.

Source: arXiv 2607.27180v2, HumanCLAW: Can Vision-Language Models Act Through a Body? (2026-08-03). Compiled by ASAP.