TutorMoments: AI Tutors Struggle to Balance Help and Independence
The Allen Institute for AI (AI2) has introduced TutorMoments, a new framework and dataset aimed at evaluating large language models (LLMs) in a tutoring context. This initiative utilizes 462 anonymized transcripts from individual math tutoring sessions for U.S. students in grades 2-7, gathered from a high-dosage program targeting Title I schools. Key decision points for tutors are annotated by 27 math educators. In TutorMoments, LLMs are tasked with tutoring for five turns during these critical moments, assessed based on scaffolding, rigor, and over-scaffolding. Testing of seven LLMs reveals a tendency to over-assist with basic prompts, while prompts that are evaluation-aware enhance their performance. Although human tutors scored lower than LLMs, they provide a naturalistic benchmark. The project also features the TutorMoments-Preview dataset and has plans for future expansion.
Key facts
- TutorMoments is a framework to evaluate LLMs' ability to balance helping and holding back in tutoring.
- The dataset includes 462 de-identified transcripts of real math tutoring sessions with U.S. students in grades 2-7.
- 27 experienced U.S.-based math teachers annotated over 1,500 key moments.
- The transcripts come from a high-dosage tutoring program serving mostly Title I schools.
- Seven LLMs were tested using two prompts: plain and evaluation-aware.
- Models tend to over-help when given only the instruction to 'tutor well.'
- The evaluation-aware prompt improves performance but does not close the gap to human tutoring.
- Human tutors scored 0.458 (appropriate scaffolding), 0.182 (appropriate rigor), and 0.496 (avoids over-scaffolding).
- The project is supported by the Gates Foundation and Learning Commons.
- The team plans to build a larger multimodal dataset and stronger scoring pipeline.
Entities
Institutions
- Allen Institute for AI (AI2)
- Gates Foundation
- Learning Commons
- Hugging Face
Locations
- United States