GRPO Convergence Requires Discrete Action Space for Small-Model Quadrotor Control
A recent arXiv study (2607.21626) explores the potential of Group Relative Policy Optimization (GRPO) in refining small language models for quadrotor continuous-control applications. When Qwen-0.5B underwent standard GRPO fine-tuning for 25 Hz quadrotor velocity control, it failed, resulting in a 0 percent success rate as entropy plummeted from 0.35 to 0.03 in just 60 steps. Although eliminating the jerk-penalty term or the KL anchor halted the entropy decline, it did not facilitate learning. In contrast, substituting the continuous action interface with a 5-way categorical selection of PID presets led to successful training. The final controller achieved 98.6 percent success with a jerk of 0.656 m/s³ at 64 steps, and 100 percent success with 1.103 m/s³ jerk (or 0.796 under a velocity cap) at 256 steps, assessed across three pretrained models.
Key facts
- Vanilla GRPO fine-tuning of Qwen-0.5B for quadrotor control collapses to zero action with 0% success rate.
- Entropy drops from 0.35 to 0.03 within 60 steps during collapse.
- Removing jerk-penalty or KL anchor prevents entropy collapse but does not enable learning.
- Discrete 5-way categorical action space over PID presets enables convergence.
- At 64 steps: 98.6% success, 0.656 m/s³ jerk.
- At 256 steps: 100% success, 1.103 m/s³ jerk (0.796 with velocity cap).
- Study uses Qwen-0.5B model and simulated quadrotor control.
- Evaluation across three pretrained models shows consistent results.
Entities
Institutions
- arXiv