TTRL-Guard Framework Addresses Majority Vote Misinterpretation in Test-Time RL

A recent study questions the understanding of accuracy improvements in test-time reinforcement learning (TTRL) concerning mathematical reasoning benchmarks. The researchers contend that the enhancements linked to majority vote pseudo-labeling often stem from refining problems that are already solvable, rather than indicating true learning. Their findings reveal that instances corrupted from correct to incorrect far exceed those that are genuinely learned, with this degradation becoming permanent once the majority vote aligns with an incorrect response. This effect is referred to as the "Correct-Answer Extinction Window," which denotes a brief period when accurate signals in low-ability problems are present before being overshadowed. The study introduces TTRL-Guard, a streamlined framework featuring three mechanisms: Flip-Rate-Aware Reward Scaling (FRS), Minority-Preserving Sampling (MPS), among others aimed at addressing the extinction window. The research is available on arXiv under ID 2605.19444.