DynaSTI: Dynamics Modeling with Sequential Temporal Information for Reinforcement Learning in Atari

Deep reinforcement learning (DRL) has shown remarkable capabilities in solving sequential decision-making problems. However, DRL requires extensive interactions with image-based environments. Existing methods have combined self-supervised learning or data augmentation to improve sample efficiency. While understanding the temporal information dynamics of the environment is important for effective learning, many methods do not consider these factors. To address the sample efficiency problem, we propose dynamics modeling with sequential temporal information (DynaSTI) that incorporates environmental dynamics and leverages the correlation among trajectories to improve sample efficiency. DynaSTI uses an effective learning strategy for state representation as an auxiliary task, using gated recurrent units to capture temporal information. It also integrates forward and inverse dynamics modeling in a hierarchical configuration, enhancing the learning of environmental dynamics compared to using each model separately. The hierarchical structure of DynaSTI enhances the stability of inverse dynamics modeling during training by using inputs derived from forward dynamics modeling, which focuses on feature extraction related to controllable state. This approach effectively filters out noisy information. Consequently, using denoised inputs from forward dynamics modeling results in improved stability when training inverse dynamics modeling, rather than using inputs directly from the encoder. We demonstrate the effectiveness of DynaSTI through experiments on the Atari game benchmark, limiting the environment interactions to 100k steps. Our extensive experiments confirm that DynaSTI significantly improves the sample efficiency of DRL, outperforming comparison methods in terms of statistically reliable metrics and nearing human-level performance.

Paper Link

Recommended citation: Kim, J., Lee, Y. J., Kwak, M., Park, Y. J., & Kim, S. B. (2024). DynaSTI: Dynamics Modeling with Sequential Temporal Information for Reinforcement Learning in Atari. Knowledge-Based Systems, 112103.