Object-Focused Advice in Reinforcement Learning
Authors: 
Samantha Krening, Brent Harrison, Karen M. Feigh, Charles Isbell, Andrea ThomazAuthors Info & Claims
Pages 1447 - 1448
Abstract
In order for robots and intelligent agents to interact with and learn from people with no machine-learning expertise, robots should be able to learn from natural human instruction. Many human explanations consist of simple sentences without state information, yet most machine learning techniques that incorporate human guidance cannot use non-specific explanations. This work aims to learn policies from a few sentences that aren't state specific. The proposed Object-focused advice links an object to an action, and allows a person to generalize over an object's state space. To evaluate this technique, agents were trained using Object-focused advice collected from participants in an experiment in the Mario Bros. domain. The results show that Object-focused advice performs better than when no advice is given, the agent can learn where to apply the advice in the state space, and the agent can recover from adversarial advice. Also, including warnings of what not do to in addition to advice of what actions to take improves performance.
References
[1]
B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469--483, 2009.
[2]
S. Chernova and A. L. Thomaz. Robot learning from human teachers. Synthesis Lectures on Artificial Intelligence and Machine Learning, 8(3):1--121, 2014.
[3]
L. C. Cobo, C. L. Isbell, and A. L. Thomaz. Object focused q-learning for autonomous agents. In Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems, pages 1061--1068. International Foundation for Autonomous Agents and Multiagent Systems, 2013.
[4]
S. Griffith, K. Subramanian, J. Scholz, C. Isbell, and A. L. Thomaz. Policy shaping: Integrating human feedback with reinforcement learning. In Advances in Neural Information Processing Systems, pages 2625--2633, 2013.
[5]
J. MacGlashan, M. Babes-Vroman, M. desJardins, M. Littman, S. Muresan, S. Squire, S. Tellex, D. Arumugam, and L. Yang. Grounding english commands to reward functions. In Proceedings of Robotics: Science and Systems, Rome, Italy, July 2015.
[6]
R. Maclin, J. Shavlik, L. Torrey, T. Walker, and E. Wild. Giving advice about preferred actions to reinforcement learners via knowledge-based kernel regression. In Proceedings of the National Conference on Artificial intelligence, volume 20, page 819. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999, 2005.
[7]
J. Togelius, S. Karakovskiy, and R. Baumgarten. The 2009 mario ai competition. In Evolutionary Computation (CEC), 2010 IEEE Congress on, pages 1--8. IEEE, 2010.
Index Terms
- Object-Focused Advice in Reinforcement Learning
Recommendations
Reinforcement learning from simultaneous human and MDP reward
AAMAS '12: Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems - Volume 1As computational agents are increasingly used beyond research labs, their success will depend on their ability to learn new skills and adapt to their dynamic, complex environments. If human users---without programming skills---can transfer their task ...
Combining manual feedback with subsequent MDP reward signals for reinforcement learning
AAMAS '10: Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: volume 1 - Volume 1As learning agents move from research labs to the real world, it is increasingly important that human users, including those without programming skills, be able to teach agents desired behaviors. Recently, the tamer framework was introduced for ...
Integrating Agent Advice and Previous Task Solutions in Multiagent Reinforcement Learning
AAMAS '19: Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent SystemsReinforcement learning methods have successfully been applied to build autonomous agents that solve challenging sequential decision-making problems. However, agents need a long time to learn a task, especially when multiple autonomous agents are in the ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
May 2016
1580 pages
ISBN:9781450342391
- General Chairs:
- Catholijn M. Jonker,
- Stacy Marsella,
- Program Chairs:
- John Thangarajah,
- Karl Tuyls
Sponsors
- IFAAMAS
In-Cooperation
Publisher
International Foundation for Autonomous Agents and Multiagent Systems
Richland, SC
Publication History
Published: 09 May 2016
Check for updates
Author Tags
Qualifiers
- Extended-abstract
Funding Sources
- ONR
Conference
AAMAS '16
Sponsor:
AAMAS '16: International Conference on Agents and Multiagent Systems
May 9 - 13, 2016
Singapore, Singapore
Acceptance Rates
AAMAS '16 Paper Acceptance Rate 137 of 550 submissions, 25%;
Overall Acceptance Rate 1,155 of 5,036 submissions, 23%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 191Total Downloads
- Downloads (Last 12 months)58
- Downloads (Last 6 weeks)13
Reflects downloads up to 15 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in