Convergence of a Q-Learning Variant for Continuous States and Actions

Research output: Contribution to journalArticlepeer-review

9 Scopus citations
1 Downloads (Pure)

Abstract

This paper presents a reinforcement learning algorithm for solving innite horizon Markov Decision Processes under the expected total discounted reward criterion when both the state and action spaces are continuous. This algorithm is based onWatkins' Q-learning, but uses Nadaraya-Watson kernel smoothing to generalize knowledge to unvisited states. As expected, continuity conditions must be imposed on the mean rewards and transition probabilities. Using results from kernel regression theory, this algorithm is proven capable of producing a Q-value function estimate that is uniformly within an arbitrary tolerance of the true Q-value function with probability one. The algorithm is then applied to an example problem to empirically show convergence as well. © 2014 AI Access Foundation. All rights reserved.
Original languageAmerican English
Pages (from-to)705-731
Number of pages27
JournalJournal of Artificial Intelligence Research
Volume49
DOIs
StatePublished - Apr 29 2014

Keywords

  • Kernel regression theory
  • Kernel smoothing
  • Markov decision processes
  • Nadaraya--Watson
  • Q-learning
  • Q-value function estimate
  • Reinforcement learning
  • State and action spaces

DC Disciplines

  • Education
  • Mathematics

Fingerprint

Dive into the research topics of 'Convergence of a Q-Learning Variant for Continuous States and Actions'. Together they form a unique fingerprint.

Cite this