Student-Centric Answer Selection Improves LLM Training Over Strongest Teacher
A new framework called Student-Centric Answer Sampling (SCAS) challenges the common practice of using the highest-performing teacher model to generate training data for student LLMs. The research, published on arXiv (2605.26872), demonstrates that even when multiple teachers produce correct answers, the strongest teacher's response is not always the best supervision for a given student. SCAS selects from verified teacher-generated answers based on estimated student-centric learning cost, derived from a token-wise gradient decomposition. Experiments across 30 teacher models and 6 student base models show improved training efficiency and effectiveness.
Key facts
- arXiv paper 2605.26872 introduces SCAS framework
- SCAS selects teacher answers based on student-centric learning cost
- Current practice uses highest-performing teacher for training data
- Strongest teacher answer is not always best for student
- Method uses token-wise gradient decomposition
- Tested across 30 teacher models and 6 student base models
- SCAS provides efficient forward-only proxy for learning cost
- Research challenges assumption that teacher test performance equals teaching quality
Entities
Institutions
- arXiv