SINKFLEX-RL: Modular RL System for Long-Horizon Tool-Use Agents

A recent preprint on arXiv (2608.10357) unveils SINKFLEX-RL, a modular framework tailored for reinforcement learning (RL) in environments requiring prolonged tool usage. This system tackles issues like multi-turn on-policy rollouts that generate extended contexts and the need for custom masks and learned sink normalization in model-specific attention layers. SINKFLEX-RL integrates a Gymnasium-compatible environment wrapper, a VERL-style dataflow for rollouts, group-relative policy optimization without relying on a separate value model, and a sink-aware FlexAttention mechanism to maintain model-specific sink scaling under both causal and sliding-window masks. In an initial Tau2Bench retail evaluation, the validation reward (mean@1) rose from 0.25 at the start of training to 0.44 later, with improvements also seen in training-score and trajectory-reward metrics. This system is proposed as an effective solution for agents needing to navigate user objectives, domain policies, tool usage, simulator states, and delayed verifiable rewards.