Introduction

Policy gradient methods are a cornerstone of modern reinforcement learning, offering a direct approach to optimizing policies through gradient ascent on expected rewards. However, they suffer from high variance in gradient estimates, which can lead to unstable learning and poor sample efficiency. This briefing explores key variance reduction techniques that have become standard practice in policy gradient algorithms.

Baseline Subtraction

The most fundamental variance reduction technique is subtracting a baseline from the return estimate. The baseline does not introduce bias if it is independent of the action, but it can significantly reduce variance. Common choices include:

• State-value function V(s)
• Average reward
• Learning a separate baseline network

Mathematically, the policy gradient theorem with a baseline b(s) becomes:

∇J(θ) = E[∇log π_θ(a|s) (Q^π(s,a) - b(s))]

When b(s) = V^π(s), this reduces to the advantage function A^π(s,a).

Generalized Advantage Estimation (GAE)

GAE provides a trade-off between bias and variance by using a weighted average of n-step advantages. It introduces a λ parameter that controls the bias-variance trade-off:

GAE(γ,λ) = Σ_{l=0}^{∞} (γλ)^l δ_{t+l}

where δ_t = r_t + γV(s_{t+1}) - V(s_t) is the temporal difference error.

GAE with λ=1 corresponds to Monte Carlo advantages (high variance, low bias), while λ=0 corresponds to one-step TD advantages (low variance, high bias). Intermediate values provide a balanced approach.

Trust Region Methods

While not strictly variance reduction techniques, trust region methods like TRPO and PPO indirectly improve stability by limiting policy updates, which allows for more aggressive use of variance reduction techniques without catastrophic policy changes.

PPO uses clipped surrogate objectives to prevent large updates, while maintaining computational simplicity.

Practical Considerations

In modern deep RL implementations:

• Actor-critic methods naturally incorporate baselines through the critic network
• GAE is the default advantage estimator in most frameworks (e.g., Stable Baselines3, RLlib)
• Normalization of advantages across batches further reduces variance
• Reward scaling and clipping can help stabilize training

Conclusion

Variance reduction is essential for practical policy gradient optimization. Combining baseline subtraction, GAE, and trust region constraints enables stable and sample-efficient training of complex policies in high-dimensional environments. Ongoing research continues to refine these techniques, particularly in the context of offline RL and meta-learning settings.