Combining Reinforcement Learning with a Local Control Algorithm

Combining Reinforcement Learning with a Local Control Algorithm
ICML-2000 ( gzipped Postscript - 134 kb )

Abstract: For reinforcement learning algorithms to find wider use as on-line methods for improving control performance of real systems it is important to devise methods that take advantage of conventional control methodologies to 1) reduce the complexity of the learning problem and 2) to provide for acceptable system behavior during learning. In this paper we explore combining reinforcement learning with a hand-crafted local controller in a manner suggested by the chaotic control algorithm of Vincent, Schmitt and Vincent (1994). A closed-loop controller is designed using conventional means that creates a domain of attraction about a target state. Chaotic behavior is induced to bring the system into this region, at which time the local controller is turned on to bring the system to the target state and stabilise it there. Here we describe experiments in which we use reinforcement learning instead of, and in addition to, chaotic behavior to learn an efficient policy for driving the system into the local controller's domain of attraction. Using a simulated double pendulum, we illustrate how this method allows reinforcement learning to be effective in a problem that cannot be easily solved by reinforcement learning alone, and we show how reinforcement learning can improve upon the chaotic control algorithm when the domain on attraction can only be approximately determined. This is a simple and effective way of extending reinforcement learning to more difficult problems.