AI tutors fail to challenge students: GPT-5.5 pushed for rigor in only 4.6% of the moments that called for it

The Allen Institute for AI released TutorMoments on August 7, 2026, a framework that measures whether AI tutors know when to help a student and when to hold back. Evaluating seven language models by replaying 462 real math tutoring transcripts and 1,536 key moments marked by 27 teachers, the report finds that under a plain tutoring prompt GPT-5.5 pushed for rigor in only 4.6% of the moments that called for it, with Claude Opus 4.8 at 20.8% and Gemini 2.5 Pro at 22.3%. Given a prompt that spells out the pedagogical trade-off, the same GPT-5.5 score rose from 0.046 to 0.531. ASAP works from Ai2's preview report and released dataset to set out what this evaluation actually measures.

Knowing how to help and knowing when to stop are separate abilities

TutorMoments targets a model's ability to withhold help rather than its ability to give it. Ai2's researchers frame the problem by noting that general language model development optimizes for behaviors like helpfulness, while an effective AI tutor must know when to help and when to challenge students. Where existing tutoring benchmarks score behaviors themselves, such as whether a tutor avoids giving away the answer or provides a scaffold, TutorMoments asks whether that behavior was appropriate for the specific learning moment.

The evaluation rests on three metrics. Appropriate scaffolding asks whether the tutor supported the student without over-scaffolding at moments teachers judged as calling for support. Appropriate rigor asks whether the tutor increased cognitive demand at moments teachers judged the student ready for it. Avoiding over-scaffolding measures, across all key moments, how often the tutor did not reduce difficulty more than necessary. All three run from 0 to 1, and over-scaffolding is penalized in every case.

The mechanism is replay. A real student-tutor transcript is cut at a point teachers identify, and a language model tutor takes over from there. The student side is played by Claude Opus 4.6, conditioned on a description of the original student's behavior, and each replay is fixed at five turns, three from the tutor and two from the synthetic student. Holding the dialogue history constant so that different instructional choices can be compared from the same pedagogical state is the core of the design.

Under plain prompts, every model failed to push for rigor

The plain-prompt results for all seven models lean in one direction. Appropriate rigor scores are 0.046 for GPT-5.5, 0.023 for GPT-5.4 mini, 0.085 for Claude Sonnet 4.6, 0.088 for DeepSeek V4 Pro, 0.158 for Gemini 3.5 Flash, 0.208 for Claude Opus 4.8, and 0.223 for Gemini 2.5 Pro. Appropriate scaffolding under the same condition runs far higher at 0.727 for Gemini 2.5 Pro, 0.723 for Gemini 3.5 Flash, and 0.615 for Claude Opus 4.8. Helping when help is called for is partly solved; holding back when holding back is called for is essentially not.

The over-scaffolding numbers support that reading. Avoiding over-scaffolding under a plain prompt is 0.173 for GPT-5.5, 0.188 for GPT-5.4 mini, 0.379 for DeepSeek V4 Pro, 0.437 for Claude Sonnet 4.6, 0.462 for Claude Opus 4.8, 0.546 for Gemini 2.5 Pro, and 0.598 for Gemini 3.5 Flash. Even the best figure means the tutor cut difficulty more than necessary in four of every ten moments, and the worst means it did so in more than eight. Ai2 describes this as frontier models defaulting toward helpfulness at the expense of adapting instruction to learners' needs.

The metrics also disagree on ranking. Gemini 2.5 Pro leads on scaffolding under a plain prompt while Gemini 3.5 Flash leads on avoiding over-scaffolding, and Claude Opus 4.8 and Gemini 2.5 Pro lead on rigor. General capability rankings do not transfer intact to this domain, though the researchers note the benchmark does recover known orderings of model size and capability while still separating frontier models from one another.

One paragraph of prompting raised GPT-5.5's rigor score by a factor of 11.5

The largest single variable in this study is the prompt rather than the model. Under an evaluation-aware prompt that describes the trade-off between scaffolding, over-scaffolding, and rigor and instructs the model to stay in the student's zone of proximal development, appropriate rigor rose from 0.046 to 0.531 for GPT-5.5, a factor of 11.5, from 0.085 to 0.792 for Claude Sonnet 4.6, a factor of 9.3, and from 0.208 to 0.831 for Claude Opus 4.8, a factor of 4.0. Avoiding over-scaffolding rose from 0.437 to 0.913 for Claude Sonnet 4.6 and from 0.462 to 0.896 for Claude Opus 4.8.

That result carries two meanings at once. The first is that failure to push for rigor was not an absence of capability. When identical weights behave nearly ten times differently after one paragraph of instruction, the low plain-prompt score is a product of default behavior policy rather than a ceiling. Models aligned to prioritize helpfulness default to over-helping in educational settings, and this contrast is what makes that visible.

The second is that the improvement does not arrive on its own. Ai2 cites prior work showing everyday users write far simpler prompts such as "Act like a tutor," and states it is unknown whether popular chat interfaces route these underspecified queries through tutoring-specialized harnesses. A gap therefore remains between the ceiling measured in the lab and the behavior a student actually meets, and that gap is prompt design. For a school or learning service attaching a language model as a tutor, the practical conclusion is that the decisive work is system prompt design rather than model selection.

Some of the gap survives prompting entirely. Even with the evaluation-aware prompt, appropriate rigor reached only 0.531 for GPT-5.5 and 0.492 for DeepSeek V4 Pro, and the researchers conclude that prompting alone is insufficient to produce consistently appropriate instructional decisions.

Misreading the human baseline of 0.458 inverts the conclusion

The most easily misread numbers in the report are the human tutor baselines of 0.458, 0.182, and 0.496. Human tutors score 0.458 on appropriate scaffolding, 0.182 on appropriate rigor, and 0.496 on avoiding over-scaffolding, all below Claude Opus 4.8's 0.858, 0.831, and 0.896 under the evaluation-aware prompt. Taken alone, those figures support a sentence claiming AI tutors have surpassed human teachers.

Ai2 blocks that reading directly in the report. The authors state that human tutors do not necessarily represent ideal tutoring practice, that teacher annotators were instructed to flag ineffective tutoring moments when composing the key moment dataset, and that the dataset therefore concentrates on human tutors' missed opportunities and miscalibrations. Human tutors serve as naturalistic reference points for contextualizing model behavior rather than as a pedagogically normative gold standard.

The data collection conditions reinforce this. Tutors at the high-dosage tutoring provider supplying the transcripts are trained volunteers or college students in a work-study program, and they received no AI assistance. The comparison group is volunteer tutors rather than expert teachers, and the scored moments are precisely the points where teachers judged those tutors to have misjudged. Reading 0.458 as the average competence of human teachers reverses the conclusion the data supports.

Situational adaptivity is the further metric to watch. Measuring how much a tutor's action distribution shifts between scaffolding-appropriate and rigor-appropriate moments by KL divergence, human tutors score 0.179 and 0.177 while plain-prompted models range from 0.048 to 0.182. Plain-prompted models largely repeat the same behavior as the situation changes, and only Claude Opus 4.8 matches the human baseline for situational sensitivity. A high aggregate score does not by itself indicate that a tutor adapts its strategy.

How the data was collected is what gives the evaluation its weight

TutorMoments rests on 462 authentic tutoring transcripts and the judgments of 27 practicing math teachers rather than on synthetic dialogue. The released TutorMoments-Preview dataset comprises 462 text transcripts between 198 U.S. students in grades 2 through 7 and 173 human tutors, averaging 384.1 turns and 53.5 minutes per session. Students and tutors met an average of 2.3 times with the same partner, and the transcripts are almost exclusively math tutoring. Teachers annotated 122 of these transcripts, marking 1,536 unique key moments.

The annotators are 27 U.S.-based teachers with more than three years of math teaching experience, 14 of them with more than ten years. Pay starts at $35 per hour, rising to $40 after five hours of completed annotation and $45 after 15 hours. For each moment, teachers wrote free-text descriptions of the situation, the tutor's action, and the result, averaging 81.5 words. Each moment received 3.5 annotations on average, and where multiple teachers commented on situation appropriateness, 68% agreed on scaffolding and 63% agreed on a rigor push.

The evaluation itself uses 520 moments: all 260 classified as rigor-appropriate plus 260 randomly drawn from the 738 classified as scaffolding-appropriate, for a 50-50 split. Scoring runs through a language model pipeline, and validation shows a scaffolding-direction F1 of 0.9313 for Claude Opus 4.8 against a markedly lower 0.6401 for rigor. The researchers attribute that gap to prior findings that rigor is difficult even for experienced instructors to identify, define, and apply consistently.

What a learning service can take from this evaluation

The transferable asset here is the evaluation method rather than the scoreboard. Ai2 released the dataset on Hugging Face and the code on GitHub, so the replay design can be applied to a service's own conversation logs to produce the same three metrics. Measuring where a tutor over-helps moment by moment, rather than through an aggregate satisfaction score, is what separates this from conventional quality surveys.

What does not transfer directly is equally clear. The data reflects U.S. students and U.S. teachers' judgments, covers mathematics only, and spans grades 2 through 7. Norms for learning and for when a teacher should intervene vary by culture and curriculum, so applying this elsewhere means repeating the annotation stage with local teachers on local transcripts. What the work makes reusable is a framework that defines the situation first and then asks about appropriateness within it, not a set of ground-truth answers.

The immediately actionable finding sits on the prompt side. A single paragraph naming the trade-off between scaffolding and rigor raised rigor scores by as much as a factor of 11.5, which quantifies the value of writing pedagogical decision criteria directly into a system prompt. Any organization evaluating AI tutors should allocate effort to designing and validating that paragraph before comparing models.

Conditions to state alongside these numbers

The first condition is scale and scope: TutorMoments-Preview covers 520 scored moments of text-based U.S. math tutoring in grades 2 through 7. This is a preview report, the dataset itself is named a preview, and the evaluation covers 520 moments of text-based U.S. math tutoring. Ai2 states it is releasing this sample to gather feedback while preparing a larger multimodal dataset and an improved scoring pipeline. Citing the table as the performance of AI tutors in general runs past the evidence.

The second condition is that the grader is a language model. The scoring pipeline runs on Claude Opus 4.8, and Claude models are also among the systems evaluated. The researchers note they developed prompts primarily with Claude and that performance only somewhat generalizes to other model families, and they cite prior work on language models preferring their own outputs while calling for tuning scoring prompts across multiple models. With rigor-direction F1 at 0.6401, rigor figures carry more uncertainty than scaffolding figures.

The third condition is that the student is not a person. The student role in replay is played by Claude Opus 4.6, and this synthetic student can see the full original transcript, making it an oracle student with information the model tutor lacks. Ai2 reports that in pilot runs even oracle synthetic students tended to produce unnaturally successful learning behaviors, and presently uses them as a placeholder for observing tutor behavior. With replays fixed at five turns, these scores measure instructional choices under controlled conditions rather than learning outcomes.

The questions this evaluation has to answer next

The largest open question TutorMoments leaves is whether appropriate instructional choices translate into learning gains. The framework measures alignment between a tutor's actions and what teachers judged appropriate; it does not measure changes in student performance or understanding. Connecting teacher judgment to learning outcomes remains separate work.

The second question is whether defaults can change. That prompting moves scores so sharply implies room to adjust helpfulness bias in post-training. How this table shifts if model providers tune default behavior for educational contexts is the thing to watch in the next revision.

The third question is transfer across language and culture. Scaffolding and rigor come from education theory, but the criteria for applying them rest on teachers' culturally situated judgment, and even here inter-teacher agreement reached only 68% and 63%. How those agreement rates move when the framework is rebuilt in another language and curriculum will determine how far this method scales.

Source: ASAP analysis based on the Ai2 preview report "When Help is Unhelpful: Evaluating AI Tutors for Productive Struggle" (August 7, 2026), the allenai/tutormoments-preview dataset on Hugging Face, and the code at github.com/allenai/tutormoments