Integral reinforcement learning and experience replay for adaptive optimal control of partially-unknown constrained-input continuous-time systems

Abstract

In this paper, an integral reinforcement learning (IRL) algorithm on an actor–critic structure is developed<br><br>to learn online the solution to the Hamilton–Jacobi–Bellman equation for partially-unknown constrainedinput<br><br>systems. The technique of experience replay is used to update the critic weights to solve an<br><br>IRL Bellman equation. This means, unlike existing reinforcement learning algorithms, recorded past<br><br>experiences are used concurrently with current data for adaptation of the critic weights. It is shown that<br><br>using this technique, instead of the traditional persistence of excitation condition which is often difficult<br><br>or impossible to verify online, an easy-to-check condition on the richness of the recorded data is sufficient<br><br>to guarantee convergence to a near-optimal control law. Stability of the proposed feedback control law is<br><br>shown and the effectiveness of the proposed method is illustrated with simulation examples