Deep Reinforcement Learning using Capsules in Advanced Game Environments

Abstract

Reinforcement Learning (RL) is a research area that has blossomed tremendously in recent years and has shown remarkable potential for arti cial intelligence based opponents in computer games. This success is primarily due to vast capabilities of Convolutional Neural Networks (ConvNet), enabling algorithms to extract useful information from noisy environments. Capsule Network (CapsNet) is a recent introduction to the Deep Learning algorithm group and has only barely begun to be explored. The network is an architecture for image classi cation, with superior performance for classi cation of the MNIST dataset. CapsNets have not been explored beyond image classi cation. This thesis introduces the use of CapsNet for Q-Learning based game algorithms. To successfully apply CapsNet in advanced game play, three main contributions follow. First, the introduction of four new game environments as frameworks for RL research with increasing complexity, namely Flash RL, Deep Line Wars, Deep RTS, and Deep Maze. These environments ll the gap between relatively simple and more complex game environments available for RL research and are in the thesis used to test and explore the CapsNet behavior. Second, the thesis introduces a generative modeling approach to produce arti cial training data for use in Deep Learning models including CapsNets. We empirically show that conditional generative modeling can successfully generate game data of su cient quality to train a Deep Q-Network well. Third, we show that CapsNet is a reliable architecture for Deep Q-Learning based algorithms for game AI. A capsule is a group of neurons that determine the presence of objects in the data and is in the literature shown to increase the robustness of training and predictions while lowering the amount training data needed. It should, therefore, be ideally suited for game plays. We conclusively show that capsules can be applied to Deep Q-Learning, and present experimental results of this method in the environments introduced. We further show that capsules do not scale as well as convolutions, indicating that CapsNet-based algorithms alone will not be able to play even more advanced games without improved scalability