Distributional Soft Actor Critic

Distributional Soft Actor Critic is a suite of model-free off-policy reinforcement learning algorithms, tailored for learning decision-making or control policies in complex systems with continuous action spaces. Distinct from traditional methods that focus solely on expected returns, DSAC algorithms are designed to learn a Gaussian distribution over stochastic returns, called value distribution. This focus on Gaussian value distribution learning notably diminishes value overestimations, which in turn boosts policy performance. Additionally, the value distribution learned by DSAC can also be used for risk-aware policy learning. From a technical standpoint, DSAC is essentially a distributional adaptation of the well-established soft actor-critic method.
To date, the DSAC family comprises two iterations: the original DSAC-v1 and its successor, DSAC-T, with the latter demonstrating superior capabilities over the Soft Actor-Critic in Mujoco benchmark tasks. The source code for DSAC-T can be found at the following URL: .
Both iterations have been integrated into an advanced, Pytorch-powered reinforcement learning toolkit named GOPS: .