Actor-critic

Actor-critic methods combine a policy, the actor, with a learned value estimator, the critic. The critic supplies a lower-variance learning signal.

Definition

Actor-critic methods combine a policy, the actor, with a learned value estimator, the critic. The critic supplies a lower-variance learning signal.

The actor chooses; the critic estimates how much better an outcome was than expected.

For reward 1, discount 0.9, next value 4, and current value 3, the one-step TD advantage estimate is 1.6.

$\delta_t=r_t+\gamma V(s_{t+1})-V(s_t)$ . A terminal transition uses zero next-state value.

Inspect critic loss and policy behavior together: an inaccurate critic can confidently push the actor in the wrong direction.

Assembled from the ReLU.chat curated knowledge base. These explanations are concise on purpose; check the sources for anything important.