Advancements in Safe Deep Reinforcement Learning for Real-Time Strategy Games and Industry Applications

Abstract

Deep reinforcement learning has attracted considerable attention from industry and academia, among others, because of its success in solving intricate video games and industrial applications. Recent advancements in hardware and computing exponentially increase computational power availability, facilitating deep neural networks training. These networks can learn the RL behavior policy from high-dimensional data and perform significantly better than exact tabular solutions, albeit requiring considerably more computer resources. Games are among the most used applications to assess reinforcement learning (RL) algorithms’ behavioral properties and planning efficiency. They can provide the data structure and volume required to train deep learning models. Specially crafted games can express real-world industry applications to reduce setup costs while drastically increasing reproducibility. RL can improve efficiency in industrial applications where expert systems dominate the scene, reduce manual and potentially dangerous labor. The problem with applied industrial reinforcement learning is that traditional methods learn by trial and error. Because of this, RL agents risk encountering catastrophic events during learning, which can cause damage to humans or equipment. Therefore, using games to train and study safe RL agents is appealing. Real-Time Strategy (RTS) games are especially captivating because of their high dimensional state and action spaces. Furthermore, RTS games share many attributes with industrial and real-world applications, such as simultaneous actions, imperfect information, and system stochasticity. Recent advancements show that model-free RL algorithms can learn superhuman performance in games such as StarCraft II, again using substantial computational power. Therefore, the downside is that these algorithms are expensive and hard to train, making it challenging to use the same methods for industrial applications. There are also substantial state-space complexity gaps in open-source environments. This restricts algorithm evaluation to only a subset of tasks required for operating sufficiently in industry applications.