Understanding Extortion and Fairness in Iterated Prisoner's Dilemma Through Actor-Critic Learning Dynamics

Iterated Prisoner's Dilemma (IPD) is widely used to investigate the emergence and stability of cooperative behavior in both social and biological systems. Through repeated interactions, cooperation can be cultivated, enhancing the long-term mutual benefit a principle fundamental to direct reciprocity. However, this mechanism also permits extortionate Zero-Determinant (ZD) strategies to exploit Always Cooperate (ALLC) and other cooperative strategies. In this study, we examine the learning dynamics of actor-critic agents when confronted with extortionate ZD strategies in IPD. We analyze the learning process of actor-critic agents in both stochastic and deterministic settings, exploring the condition under which cooperation can emerge and persist in the presence of extortionate ZD strategies. Furthermore, we scrutinize the balance between fairness and exploitation, examining how extortionate ZD strategies can maximize their rewards without destabilizing cooperative equilibria. Our results offer valuable insights into the reinforcement learning dynamics in the context of IPD and illuminate the interplay between evolutionary game theory and reinforcement learning in understanding the emergence of cooperation, paving the way for developing resilient, cooperative multi-agent systems.