State–action–reward–state–action

This name reflects the fact that the main function for updating the Q-value depends on the current state of the agent "S₁", the action the agent chooses "A₁", the reward "R₂" the agent gets for choosing this action, the state "S₂" that the agent enters after taking that action, and finally the next action "A₂" the agent chooses in its new state. The acronym for the quintuple (S_t, A_t, R_t+1, S_t+1, A_t+1) is SARSA. Some authors use a slightly different convention and write the quintuple (S_t, A_t, R_t, S_t+1, A_t+1), depending on which time step the reward is formally assigned. The rest of the article uses the former convention.

Algorithm

${\displaystyle Q^{\text{new}}(S_t,A_t)\leftarrow (1-\alpha )Q(S_t,A_t)+\alpha [R_{t+1}+\gamma Q(S_{t+1},A_{t+1})]}$

A SARSA agent interacts with the environment and updates the policy based on actions taken, hence this is known as an on-policy learning algorithm. The Q value for a state-action is updated by an error, adjusted by the learning rate α. Q values represent the possible reward received in the next time step for taking action a in state s, plus the discounted future reward received from the next state-action observation.

Watkin's Q-learning updates an estimate of the optimal state-action value function ${\displaystyle Q^{\ast }}$ based on the maximum reward of available actions. While SARSA learns the Q values associated with taking the policy it follows itself, Watkin's Q-learning learns the Q values associated with taking the optimal policy while following an exploration/exploitation policy.

Some optimizations of Watkin's Q-learning may be applied to SARSA.