Online Concurrent Reinforcement Learning Algorithm to Solve Two-player Zero-sum Games for Partially-unknown Nonlinear Continuous-time Systems

Abstract

Online adaptive optimal control methods based on reinforcement learning algorithms typically need to check for the persistence of excitation condition, which is necessary to be known a priori for convergence of the algorithm. However, this condition is often infeasible to implement or monitor online. This paper proposes an online concurrent reinforcement learning algorithm (CRLA) based on neural networks (NNs) to solve the H1 control problem of partially unknown continuous-time systems, in which the need for persistence of excitation condition is relaxed by using the idea of concurrent learning. First, H1 control problem is formulated as a two-player zero-sum game, and then, online CRLA is employed to obtain the approximation of the optimal value and the Nash equilibrium of the game. The proposed algorithm is implemented on actor–critic–disturbance NN approximator structure to obtain the solution of the Hamilton–Jacobi–Isaacs equation online forward in time. During the implementation of the algorithm, the control input that acts as one player attempts to make the optimal control while the other player, that is, disturbance, tries to make the worstcase possible disturbance. Novel update laws are derived for adaptation of the critic and actor NN weights. The stability of the closed-loop system is guaranteed using Lyapunov technique, and the convergence to the Nash solution of the game is obtained. Simulation results show the effectiveness of the proposed method.