Abstract: Momentum is a classical tool for accelerating convex optimization and a standard component of modern large language model training. Yet recent empirical work has questioned the usefulness of first-moment momentum in reinforcement learning with verifiable rewards (RLVR). We study this question through a bandit formulation of RLVR with bias-corrected exponential moving average (EMA) momentum, where the EMA coefficient β controls how quickly the weights on past gradients decay. When policy gradients are evaluated exactly, we prove that increasing β within a range that prevents overshoot strictly accelerates optimization. When gradients are estimated from sampled responses, a positive expected gain can be obscured by sampling fluctuations. We show that as the learning rate η decreases, the expected gain with fixed β vanishes faster than these fluctuations, whereas increasing β to keep η/(1-β) within the safe regime preserves a nonvanishing gain that eventually dominates them. Bandit simulations quantitatively validate these predictions. RLVR experiments on Qwen3-4B-Base using SGD with EMA and AdamW support our principle of choosing larger coefficients in practical LLM post-training. On Qwen2.5-32B with AdamW, simply increasing β₁ improves mean validation accuracy by nearly 2 percentage points over the conventional choice, demonstrating the principle's effectiveness at a larger model scale.
What is Momentum in RL?
In reinforcement learning, especially when fine-tuning large language models with verifiable rewards (like correctness checks), we often use gradient-based optimizers. Momentum helps by accumulating past gradients to smooth out updates and accelerate learning in the right direction.
The key insight from the paper: Momentum helps when the learning rate is small enough and the momentum coefficient is large enough to keep the effective step size η/(1-β) in a stable range.
Why Momentum Can Help
When gradients are noisy (from sampling), plain SGD can zigzag. Momentum averages past gradients, reducing variance and creating a more stable direction.
The Trade-off
Too much momentum can cause overshoot and instability. The paper shows there's a "safe regime" where β can be increased to preserve gains even as learning rate decreases.
Practical Takeaway
For LLM post-training with verifiable rewards, using larger momentum coefficients (like in AdamW's β₁) can give meaningful improvements — up to 2% validation accuracy in their experiments.