[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
10.5555/2615731.2616057acmotherconferencesArticle/Chapter ViewAbstractPublication PagesaamasConference Proceedingsconference-collections
poster

Monte carlo bayesian hierarchical reinforcement learning

Published: 05 May 2014 Publication History

Abstract

In this paper, we propose to use hierarchical action decomposition to make Bayesian model-based reinforcement learning more efficient and feasible in practice. We formulate Bayesian hierarchical reinforcement learning as a partially observable semi-Markov decision process (POSMDP). The main POSMDP task is partitioned into a hierarchy of POSMDP subtasks; lower-level subtasks get solved first, then higher-level ones. We sample from a prior belief to build an approximate model for each POSMDP, then solve using Monte Carlo Value Iteration with Macro-Actions solver. Experimental results show that our algorithm performs significantly better than that of flat BRL in terms of both reward, and especially solving time, in at least one order of magnitude.

References

[1]
H. Bai, D. Hsu, W. S. Lee, and N. A. Vien. Monte carlo value iteration for continuous-state pomdps. In WAFR, pages 175--191, 2010.
[2]
T. G. Dietterich. Hierarchical reinforcement learning with the maxq value function decomposition. J. Artif. Intell. Res. (JAIR), 13:227--303, 2000.
[3]
M. Duff. Optimal learning: Computational procedures for Bayes-adaptive Markov decision processes. PhD thesis, University of Massachusetts Amherst, 2002.
[4]
A. Guez, D. Silver, and P. Dayan. Efficient bayes-adaptive reinforcement learning using sample-based search. In NIPS, pages 1034--1042, 2012.
[5]
Z. W. Lim, D. Hsu, and L. W. Sun. Monte Carlo value iteration with macro-actions. In NIPS, pages 1287--1295, 2011.
[6]
J. Pineau. Tractable Planning Under Uncertainty: Exploiting Structure. PhD thesis, Robotics Institute, Carnegie Mellon University, 2004.
[7]
W. H. Turkett. Robust Multiagent Plan Generation and Execution with Decision Theoretic Planners. PhD thesis, Department of Computer Science and Engineering, University of South Carolina, 1998.
[8]
N. A. Vien and W. Ertel. Monte carlo tree search for bayesian reinforcement learning. In ICMLA (1), pages 138--143, 2012.
[9]
Y. Wang, K. S. Won, D. Hsu, and W. S. Lee. Monte carlo bayesian reinforcement learning. In ICML, 2010.

Cited By

View all
  • (2017)An efficient approach to model-based hierarchical reinforcement learningProceedings of the Thirty-First AAAI Conference on Artificial Intelligence10.5555/3298023.3298089(3583-3589)Online publication date: 4-Feb-2017
  • (2014)Approximate planning for bayesian hierarchical reinforcement learningApplied Intelligence10.1007/s10489-014-0565-641:3(808-819)Online publication date: 1-Oct-2014

Index Terms

  1. Monte carlo bayesian hierarchical reinforcement learning

    Recommendations

    Comments

    Please enable JavaScript to view thecomments powered by Disqus.

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    AAMAS '14: Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems
    May 2014
    1774 pages
    ISBN:9781450327381

    Sponsors

    • IFAAMAS

    In-Cooperation

    Publisher

    International Foundation for Autonomous Agents and Multiagent Systems

    Richland, SC

    Publication History

    Published: 05 May 2014

    Check for updates

    Author Tags

    1. Bayesian RL
    2. einforcement learning
    3. hierarchical BRL

    Qualifiers

    • Poster

    Conference

    AAMAS '14
    Sponsor:

    Acceptance Rates

    AAMAS '14 Paper Acceptance Rate 169 of 709 submissions, 24%;
    Overall Acceptance Rate 1,155 of 5,036 submissions, 23%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)2
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 10 Dec 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2017)An efficient approach to model-based hierarchical reinforcement learningProceedings of the Thirty-First AAAI Conference on Artificial Intelligence10.5555/3298023.3298089(3583-3589)Online publication date: 4-Feb-2017
    • (2014)Approximate planning for bayesian hierarchical reinforcement learningApplied Intelligence10.1007/s10489-014-0565-641:3(808-819)Online publication date: 1-Oct-2014

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media