Trust Region Policy Optimization¶

When using policy gradients, small changes in the parameter space can sometimes have very large differences in performance. This makes it dangerous to use large step sizes as a single bad step can collapse policy performance.

The trust region policy optimization algorithm aims to update policies by taking the largest step possible while satisfying a constraint on how close the new and old policies are.

$$\underset{\theta }{maximize}L(\theta ,{\theta }_{old})subjectto{D}_{KL}\left(\theta ||{\theta }_{old}\right)\le \delta$$

The surrogate advantage function $L$ measures how a policy $\pi$ performs relative to an old policy ${\pi }_{old}$.

$$L(\theta ,{\theta }_{old})=E\left[\frac{{\pi }_{\theta }\left(a|s\right)}{{\pi }_{{\theta }_{old}}\left(a|s\right)}{A}_{{\theta }_{old}}\right(s,a\left)\right]$$

Updates are constrained by the KL divergence, which measures how different a probability distribution $P$ is from another distribution $Q$.

$$D_{KL}\left(P||Q\right)=\sum _{x\in X}P\left(x\right)log\left(\frac{P\left(x\right)}{Q\left(x\right)}\right)$$

In this case, we measure the KL divergence between the new and old policy.

$$D_{KL}\left(\theta ||{\theta }_{old}\right)=\left[D_{KL}\right({\pi }_{{\theta }_{old}}(\cdot |s)||{\pi }_{\theta }(\cdot |s)\left)\right]$$

The bound for the KL divergence constraint is an arbitrary constant $\delta$.