GROW: New Reinforcement Learning Method for Flow-Matching Text-to-Speech
A novel reinforcement learning technique named GROW (Group-Relative Advantage-Weighted On-Policy RL) has been introduced for flow-matching text-to-speech (TTS) systems. This approach, outlined in an arXiv paper (2608.03215), tackles the difficulties of implementing RL in TTS when deterministic ODE sampling is utilized. Unlike conventional trajectory-level policy-gradient methods that transform the ODE into an SDE and monitor likelihood ratios, GROW directly focuses on the flow-matching objective. It samples a group of on-policy utterances for each prompt, normalizes the rewards for intelligibility and speaker similarity, and integrates them to adjust flow-matching regression. A Wasserstein-2 velocity penalty ensures the updated model remains aligned with a frozen pretrained reference, while a group-mean reward baseline facilitates advantage weighting. This method aims to enhance TTS efficiency without relying on stochastic perturbations.
Key facts
- GROW is a group-relative advantage-weighted on-policy RL method for flow-matching TTS.
- It acts directly on the standard flow-matching objective, avoiding SDE conversion.
- It samples a group of on-policy utterances per prompt and standardizes rewards.
- It combines intelligibility and speaker-similarity rewards.
- A Wasserstein-2 velocity penalty anchors the model to a frozen pretrained reference.
- A group-mean reward baseline converts reward weighting into advantage weighting.
- For strong pretrained TTS models, positive exponential weighting is dominated by reward-agnostic self-imitation.
- The paper is available on arXiv with ID 2608.03215.
Entities
Institutions
- arXiv