Optimal Output Regulation for Model-Free Quanser Helicopter with Multistep Q-Learning

In this paper, the optimal output regulation problem is considered for the model-free 2-degree-of-freedom (2-DOF) helicopter. A multistep Q-learning (MsQL) method is developed with multistep policy evaluation. First, by introducing the Q-function, the optimal output regulation problem is converted to finding the optimal Q-function. Therefore, the MsQL algorithm is proposed and its convergence theory is established by showing that it generates a nonincreasing Q-function sequence that converges to the optimal Q-function. In the MsQL, the step-size of multistep policy evaluation can be different at each iteration and an adaptive tuning rule is proposed. The MsQL learns the optimal Q-function by using real system data rather than using a system model. Finally, the developed MsQL method is employed to solve the optimal output regulation problem of the model-free 2-DOF helicopter, and its effectiveness is verified.