Towards a Relational Tsetlin Machine in Natural Language Processing

Abstract

As Artificial Intelligence (AI) becomes an integral part of everyday life both at the personal and at the societal level, there is a concerted effort to have AI models explain their decisions. Explainable Artificial Intelligence (XAI) aims to increase user trust in AI systems, as well as prevent misuse and perpetuation of bias. One of the ways that humans interact with AI is through natural language text. Since language understanding at the human level requires logical structures, integration of logic programming in natural language processing can be advantageous for natural language processing at the computational level. Due to the multifaceted nature of language, AI language systems have to consider multiple different aspects of each single piece of text. Introducing compositionality via relational modelling can capture such complex information as an aggregation of simpler parts. Like other branches of AI, Natural Language Processing (NLP) also benefits from XAI, where practitioners and end users can confirm that important information or context is not being lost in translation. Tsetlin Machines (TMs) use learning automata to provide interpretable decisions to classification problems. It learns clauses or sub-patterns constructed from the features available to it. In a simple classification task, multiple of these clauses vote to indicate which class a sample belongs to. TMs’ pattern recognition approach have proved successful in variety of image classification and NLP tasks. However, there has been no targeted research into using TMs as a tool of XAI within the aspects of language analysis. In this thesis, using terminology from XAI, we establish that the clauses learnt by a TM, taken collectively, encompass the global description of the task to be solved, and the subset of clauses that decide on a single test sample form a local description of the sample. We then establish that a TM-based system can produce human-interpretable decisions in dialogue-related semantic tasks, including entity identification and semantic relation identification. By comparing with available expert annotations, we document that the global descriptions match to a large degree on such tasks. The local descriptions allow for observation of how the model shifts its focus between different aspects of the text as required. We also exhibit that the TM can build a logical structure for reasoning based on the relationships present in natural language text. While a standard or vanilla TM depends on propositional input and creates propositional clauses to encode its decisions, we present a modified version termed as the Relational Tsetlin Machine (RTM). The RTM works on relations and their included entities, such that theresultant learning is in terms of roles played by entity types. In contrast to the vanilla TM, which uses constants or words from the vocabulary, the RTM uses variables, which allows for creation of generic Horn Clauses that effectively capture logical interactions in the text. The third contribution of this thesis is in the form of a framework that utilizes the TM to overcome the challenges of data changes. Most AI applications require large amount of data in order to perform well. But data can have different characteristics when it comes from different sources or different time instances. This is true for natural language data as well, since language changes both historically and geographically, and even between spoken, written, or on-line usage. We show that a TM-based system can identify differing characteristics in data, isolate the samples that do not conform with the majority and also provide ways to mitigate the effect such samples have on the model performance.