| dc.description.abstract | The Transformer deep learning model has recently proven its superiority in tasks like natural language processing and computer vision, as tools like ChatGPT and DALL-E have become widespread and helps humans complete tasks faster with high accuracy. This comes from the ability of Transformer models to comprehend sequential data by weighing the importance of each token in sequences through an attention mechanism and being trained on massive amounts of data. As researchers seek to apply Transformer models to other disciplines, the sequential nature of reinforcement learning tasks becomes an interesting study area. Despite the demonstrated superiority of transformer-based models in various domains, their adoption within reinforcement learning paradigms, particularly within game-based learning environments, has yet to become widespread. In particular, reinforcement learning problems where an intelligent agent learns how to act in a video game are interesting, as they can help simulate real-life scenarios and therefore make autonomous systems less expensive and safer to train. Real-Time Strategy games are complex video games where players must develop a strategy in real-time to gain an advantage over other players, and reaching game objectives often involve performing a specific sequence of actions, making them an excellent area of study for reinforcement learning combined with Transformers.
This thesis explores, evaluates and improves Transformer models applied in Real-Time Strategy Game environments with a particular focus on limited data and computational power resources. To this end, DeepRTS is chosen as the reinforcement learning environment for its high performance and simplified game mechanics, but also to enrich its relatively small research domain. This work implements several sub-environments in DeepRTS with various objectives and levels of complexity to give agents a diverse range of tasks and to compare deep learning algorithms. The authors of this thesis also contributed to the DeepRTS project by fixing source code issues to improve performance. As there is no publicly available dataset for DeepRTS to train a Transformer model on, this thesis proposes a novel model, namely the Genetic Algorithm Decision Transformer, a new implementation for data generation in reinforcement learning environments by leveraging the autoregressive Decision Transformer model for action prediction. The novelty lies in using the genetic algorithm to select the best data samples from a pool to train a Decision Transformer agent. Results are compared against a Double Deep Q-learning agent and a standard Decision Transformer agent, the latter being trained using different datasets, and results show its dependency on high-quality data. Genetic Algorithm Decision Transformer improves the aforementioned algorithms by generating its own dataset with high-quality data samples while using the same underlying Decision Transformer model. Results show that Genetic Algorithm Decision Transformer outperforms its counterpart Decision Transformer algorithm by a magnitude of up to 3.3 times the reward. However, improvements to data collection could improve the model further. | |
