SARSA

SARSA updates an action-value estimate using the next action actually chosen by the behavior policy. It is an on-policy TD control method.

Definition

SARSA updates an action-value estimate using the next action actually chosen by the behavior policy. It is an on-policy TD control method.

Because the target includes exploratory actions, the learner accounts for the risks of its own exploration.

In a cliff-walking task, exploratory moves near the edge can make SARSA prefer a safer route during training.

$Q(s,a)\leftarrow Q(s,a)+\alpha[r+\gamma Q(s^{\prime},a^{\prime})-Q(s,a)]$ , where $a^{\prime}$ is sampled from the behavior policy.

Compare SARSA and Q-learning under the same exploration schedule to separate behavior-policy effects from implementation differences.

Assembled from the ReLU.chat curated knowledge base. These explanations are concise on purpose; check the sources for anything important.