Tencent Hunyuan brings the classic critical point size theory into large model reinforcement learning PPO, with throughput increased by up to 2.29 times and GRPO saving 29% training time

Reinforcement learning for large models is heading towards larger GPU clusters and thicker training data, with training efficiency now becoming a top priority. The latest research from the Tencent Huan Yuan team focuses on an overlooked problem: when models generate their own training data and the rollout generation and training scale at different paces, how should the batch size be redefined.

The study starts from the classic theory of critical batch size and re-derives it for this new scenario of online large language model reinforcement learning. The conclusion is practical: in two mainstream algorithms, GRPO and PPO, by adjusting the learning rate within a bounded range of increased batch size, we can ensure that "each response" learns what it should learn without being diluted. In other words, the batch size is not always better when larger, nor is it fixed—it exists within a clear sweet spot that can be fine-tuned.

In terms of real hardware costs, the benefits are significant. With a fixed hardware configuration, increasing the batch size can boost the throughput of the PPO generation phase by up to 2.29 times; while the best GRPO configuration measured achieves the same validation target in 29% less time than before. For training teams that constantly burn through GPUs, this means either finishing the task faster with the same cluster and goal, or processing more within the same time frame—effectively squeezing out the excess costs from the most expensive part of reinforcement learning generation.

The significance of this research lies in providing an actionable knob for scaling online RL: in reinforcement learning, the model acts as both the student and the question setter. The mismatch between the scaling of generation and training is the source of the efficiency black hole, and the combination of critical batch size and learning rate adjustment precisely fills this gap. While the industry is still competing over who has the largest model, Tencent Huan Yuan is sharpening its knife to make the existing hardware run more cost-effectively.