Learning to Play Koi-Koi Hanafuda Card Games with Transformers

Card games are regarded as an idealized model for many real-world problems for their rich hidden information and strategic decision-making process. It provides a fertile environment for artificial intelligence (AI), especially reinforcement learning (RL) algorithms. With the boom of deep neural networks, increasing breakthroughs have been made in this challenging domain. Koi-Koi is a traditional two-player imperfect-information playing card game. However, due to its unique deck and complex rules, related researches are mostly based on handcrafted features and the custom network architecture. In this article, we design a more general AI framework, relying a transformer encoder as the network backbone with tokenized card state input, which is trained by the Monte-Carlo RL with phased round reward. Experimental results show that our AI achieves a winning rate of 53% and +2.02 average difference point versus experienced human players in multiround Koi-Koi games. Moreover, with the aid of attention mechanism, we provide a novel view for analyzing the playing strategy. Such framework design can be applied to various card games.