Three-party evolutionary game-based analysis and stability enhancement of improved PBFT consensus mechanism

Within the intricate dynamics of a blockchain system, participants with bounded rationality may engage in dishonest behavior, thereby disrupting the consensus of the system in the absence of incentive mechanisms. This vulnerability is particularly common in voting-based consensus mechanisms like Practical Byzantine Fault Tolerance (PBFT), yet existing analyses often overlook the rational perspectives of participants. In this paper, we introduce an evolutionary game-theoretic approach to extend the current incentive mechanisms, enhancing the system’s stability and security. In permissioned blockchain environments, we developed a three-party evolutionary game model involving proposers, validators and auditors that dynamically assesses node strategies under various economic incentives and attack scenarios on an improved mechanism basis. Using replicator dynamic equations, we identified the conditions under which nodes converge to different evolutionarily stable strategies. Through simulation experiments, we validated the incentive mechanism’s effectiveness in promoting honest participation and resisting bribery attacks. When dishonest nodes do not exceed 80%, the system can quickly evolve to the honest state and can cope with changes in the payoff for malfeasance, whereas the original mechanism could not reach the ideal state under any condition. This achievement is attributed to the design of the mechanism and the judicious adjustment of reward and punishment parameters. The research contributes valuable findings to the discourse on consensus security. It provides key insights for developing more robust and rational incentive mechanisms in future protocols.