Deep Q-Learning¶

Consider tasks in which an agent interacts with an environment $E$ and the goal is to select actions in a way that maximizes future rewards.

The optimal action-value function is defined as the maximum expected return achievable after seeing some sequence $s$ and then taking some action $a$, where $Q^{\ast }(s,a)=ma{x}_{\pi }E[R_t|s_t=s,a_t=a,\pi ]$.

If the optimal value $Q\ast (s^{\prime },a^{\prime })$ of the sequence $s^{\prime }$ at the next timestep was known for all possible actions $a^{\prime }$, then the optimal strategy is to select the action that maximizes the expected value.

$$Q^{\ast }(s,a)=E_{s^{\prime }\sim E}[r+\gamma \underset{a^{\prime }}{max}{Q}^{\ast }(s^{\prime },a^{\prime })|s,a]$$

In practice, estimating the action-value function is impractical because it needs to be evaluated separately for each sequence. Instead, a neural network can be used as a Q-function approximator where it is trained by minimizing a sequence of loss functions for each iteration $i$.

$$L_i\left({\theta }_i\right)=E\left[\right(y_i-Q(s,a;{\theta }_i){)}^2]$$

Differentiating the loss with respect to the weights gives:

$${\nabla }_{{\theta }_i}{L}_i\left({\theta }_i\right)=E_{s,a\sim \rho (\cdot );s^{\prime }\sim E}\left[(r+\gamma \underset{a^{\prime }}{max}Q(s^{\prime },a^{\prime };{\theta }_{i-1})-Q(s,a;{\theta }_i\left)\right){\nabla }_{{\theta }_i}Q\right(s,a;{\theta }_i\left)\right]$$

Experience replay is used to store the agent's experience at each time step $e_t=(s_t,a_t,r_t,s_{t+1})$ in a dataset $D=[e_1,...,e_n]$, pooled over many episodes into a replay memory. Q-Learning updates are applied to minibatch samples of experience drawn at random from this pool of experiences. The agent then selects an action according to an $ϵ$-greedy policy.