Each notebook contains the content and code-along of each session. We recommend that you run the notebooks from Google Colaboratory for minimal setup requirements. Edit the Fill in The Code section for coding assigments and check with our way of solving them in solutions.
Markov Decision Processes / Discrete States and Actions
- What is Reinforcement Learning: Pavlov's kitties
- How Useful is Reinforcement Learning: games, robotics, ads biddings, stock trading, etc.
- Why is Reinforcement Learning Different: level of workflow automation in classes of machine learning algorithm
- Use cases for reinforcement learning
- Reinforcement Learning Framework and Markov Decision Processes
- GridWorld example to explain:
- Problems: Markov decision processes, states, actions, and rewards
- Solutions: policies, state values, (state-)action values, discount factor, optimality equations
- Words of Caution: a few reasons Deep Reinforcement Learning Doesn't Work Yet
- Challenges:
- Read up on Bellman's equations and find out where they hid in our workshop today.
- What are you ideas about how we can find the policy policy?
- Play around with Gridworld. Tweak these variables and see what happens to state and action values:
- Expand the grid and/or add some more traps
- Wind probability
- Move rewards
- Discount factor
- Epsilon and how to decay it (or not)
Discrete States and Actions
- Blackjack-v0 environment, human play and computer play
- Optimal Strategy for Blackjack
- What is Monte Carlo Method
- Monte Carlo Prediction
- Monte Carlo Control: All-visit, First-visit, and GLIE
- Challanges:
- What are some other ways of solving reinforcement learning problems? How are they better or worse than Monte Carlo methods e.g. performance, data requirements, etc.?
- Solve at least one of the following OpenAI gym environments with discrete states and actions:
- FrozenLake-v0
- Taxi-v2
- Blackjack-v0
- Any other environments with discrete states and actions at OpenAI Gym
- Check
session2b.ipynbif you are interested in using Monte Carlo method to solve Grid World. This will give you more insight into difference between all-visit and first-visit Monte Carlo.
Discrete States and Actions
- OpenAI Gym toy environment to explain temporal difference learning: sarsa, q-learning, expected sarsa
- Homework: solve an environment with discrete states and actions such as:
- FrozenLake-v0
- Taxi-v2
- Blackjack-v0
- Take-home Challenges: Solve an environment with continuous states: discretization, tile codings, etc. such as
- Acrobat-v1
- MountainCar-v0
- CartPole-v0
- LunarLander-v2
- Points to consider:
- What are the state space, action space, and rewards of the environment?
- What algorithms did you use to solve the environment and why?
- How many episodes did you solve it in? Can you improve the performance? (Tweaking discount factor, learning rate, using Monte Carlo instead of TD)
- Tensor operations
- Feedforward
- Activation functions
- Losses
- Backpropagation
- Why is deeper usually better? Spiral example
Continuous States and Discrete Actions
- Some approaches to continuous states: discretization, tile coding, other encoding, linear approximations
- Vanilla DQN: experience replay and target functions
- Take-home Challenges: Work on an Atari game and detail the process of hyperparameter tuning
Continuous States and Discrete Actions
- Rainbow
- Vanilla DQN (experience replay + target network)
- Double DQN
- Prioritized experience replay
- Dueling networks
- Multi-step learning
- Distributional RL
- Noisy networks
- Take-home Challenges: Implement Rainbow and compare it to your last project
Continuous States and Actions
- Policy gradient methods: a2c, a3c, ddpg, REINFORCE
- Monte Carlo tree search
- Explore vs exploit: epsilon greedy, ucb, thompson sampling
- Reward function setting
- Hackathon nights to play Blackjack, Poker, Pommerman, boardgames and self-driving cars
- Sutton and Barto
- Stanford CS234
- Udacity RL Nanodegree
- David Silver Lectures
- UC Berley Lectures
- Siraj's Move 37
- Denny Britz Repo
- Intro to RL in Trading
- Hands-On Reinforcement Learning With Python
- OpenAI Gym - a toolkit for developing and comparing reinforcement learning algorithms
- Unity ML-Agent Toolkit - an open-source Unity plugin that enables games and simulations to serve as environments for training intelligent agents
- Holodeck - a high-fidelity simulator for reinforcement learning built on top of Unreal Engine 4
- AirSim - a simulator for drones, cars and more, built on Unreal Engine
- Carla - an open-source simulator for autonomous driving research
- Pommerman - a clone of Bomberman built for AI research
- MetaCar - a reinforcement learning environment for self-driving cars in the browser
- Boardgame.io - a boardgame environment
- Unity ML-Agent Toolkit - an open-source Unity plugin that enables games and simulations to serve as environments for training intelligent agents
- SLM Labs - a modular deep reinforcement learning framework in PyTorch
- Dopamine - a research framework for fast prototyping of reinforcement learning algorithms
- TRF - a library built on top of TensorFlow that exposes several useful building blocks for implementing Reinforcement Learning agent