Reinforcement Learning with Tensorflow and Keras

Day 1:

Introduction to Reinforcement Learning (20 min)

• In this lecture, we will cover some examples and use cases of Reinforcement Learning, both in practice and in (applied) research as well as provide the motivation and fundamentals for the following lectures.
  • Where can we find RL in our daily lives?
  • Problem definition
  • History and motivation of the field. Outline of research frontiers and state of the art.

From Markov chains to Markov Decision Processes (40 min)

• This lecture provides the theoretical foundation to understand the rest of the course. It is important to at least grasp the main ideas, as it will help you make the most out of this course.
  • Markov chains: definition and examples
  • Markov Decision Processes: definition and examples
  • The Dynamic programming principle
  • Value and policy iteration
  • A stochastic approximation perspective to learning

RL as a black-box optimization problem (20 min)

• We consider Reinforcement Learning as a “black-box” optimization problem and apply different approaches to it: cross entropy method and natural evolution strategies. These approaches are conceptually simple yet very powerful and competitive against more sophisticated methods.
  • Black-box algorithms: random search, genetic algorithms, and other heuristics
  • The Cross-Entropy method
  • Natural Evolution strategies

Temporal Difference Methods. Q-Learning and Sarsa. Eligibility traces (40 min)

• In this session, we will go through the different methods for estimating value functions, which are used later for estimating the optimal behavior.
  • Q-function: definition and examples
  • Estimating Q-functions via Q-Learning
  • Sarsa: Off-policy methods
  • Beyond Q-Learning: double Q-Learning, Zap Q-Learning, and others
  • Eligibility traces: forward and backward perspective

Practice: Solving Cart-Pole and MountainCar (60 min)

• We will practice our hard-earned knowledge with two fun and challenging tasks. We will compare different algorithms and see “live” their advantages and disadvantages.
  • Solve Cart-Pole and MountainCar using different methods
  • Instructor support and solutions will be provided in the end

Deep Reinforcement Learning (40-50 min)

• Building on the temporal difference methods lecture, we will show how to use state-of-the-art methods in artificial intelligence (deep learning) to calculate value functions.
  • Function approximation methods
  • Linear functions approximation methods
  • Using a multilayer perceptron for function approximation
  • Experience replay
  • Improving baseline Deep Q-Learning
  • Deep Cross-Entropy Method: a deep learning approach for black-box reinforcement learning

Wrap up and lessons learned (10-20 min)

• We will summarize the learnings of the first day and suggest how you can apply your newly acquired knowledge in a number of settings.
  • More detailed example applications
  • Suggestions for projects: build your portfolio

Day 2:

Policy methods (DDPG and TRPO) and Actor-Critic methods (60 min)

• In this session, we will consider a new class of algorithms: policy methods. These methods are more suitable for complicated tasks such as robotic arm manipulation, as they do not rely on computing value functions in advance.
  • Introduction to Policy Gradients: finite differences
  • Use likelihood ratios for improving the quality of the policy
  • REINFORCE: a Monte Carlo approach for policy approximation
  • Improving policy methods with a baseline: actor-critic methods

Practice: Solving Cliffwalk (30 min)

• We will apply the methods to solve a challenging environment. Although this is a toy problem, it exhibits a number of features characteristic of more complicated setups. Once you can debug your algorithms here, you can apply them to more complicated tasks.
  • Implementation of several algorithms to solve a simple environment:
    • REINFORCE
    • REINFORCE with baseline
    • Policy Gradients (different algorithms)

Practice: Solving Pong (30 min)

• In this session, we will tackle the problem of teaching a bot to play the classic game Pong. We will use this as an excuse to practice the policy methods we learned before.
  • Video frame pre-processing pipeline: reading and combining video frames
  • Implementation of a policy gradient algorithm using Numpy
  • Policy gradients using Keras
  • Testing different algorithms at once: using OpenAI baselines

Introduction to RL for Robotics (30 min

• In this session, we will provide an overview of how reinforcement learning is used in robotics. This includes roughly two scenarios: teaching a robot to imitate what a human lecturer does (through virtual reality) and teaching a robot how to discover the correct behaviour through trial and error.
  • The robotics control problem: definition and challenges
  • Imitation learning
  • Model-based methods

Efficient Algorithms for Robotics (30 min)

• Reinforcement Learning can be quite data-hungry, and in this session, we will explore methods to reduce the number of simulations needed to obtain high-quality policies.
  • Imitation Learning algorithms
  • Estimating the dynamics of the system via Gaussian Processes

Practice: Humanoid robot control through RL in a simulator (60 min)

• To conclude, we will apply our hard-earned knowledge to train a humanoid robot (NAO) to learn from simulations. If you have access to NAO, the optimal policy discovered can be deployed to the real robot.
  • Set up your environment: V-REP simulator and NAOQi Python development kit
  • Using V-REP as an OpenAI Gym environment
  • Training NAO robot with reinforcement learning
  • Where to go from here? Ideas for projects