Brief Course Description

This course will introduce the student to reinforcement learning. The bulk of what we will cover comes straight from the second edition of Sutton and Barto's book, Reinforcement Learning: An Introduction. However, we will also cover additional material drawn from the latest deep RL literature. The following list of topics is subject to revisions.

  1. Problem formulation
    • The reinforcement learning problem, anatomy of an RL agent
    • Bandit problems
    • Markov decision processes
  2. Value function methods
    • Monte Carlo Methods
    • Dynamic Programming
    • Temporal Difference Learning
    • Combining planning and learning
    • Monte Carlo Tree Seach (MCTS)
  3. Function Approximation
    • Linear function approximation
    • Deep neural networks and DQN
  4. Policy Gradient Methods
    • Policy gradient formulation
    • REINFORCE
    • Actor critic methods
    • Deep policy gradient and actor critic methods (e.g., DDPG, PPO)

The course schedule is subject to change. See the schedule tab above.

Textbook

Reinforcement Learning: An Introduction, 2nd Ed., Sutton and Barto. A pdf is available on the website or you can purchase a hardcopy.

Prerequisites/Corequisites

  1. Python 3, Pytorch: You must be fluent in writing Python and able to manage Python packages. You should be able to do auto-differentiation in Python such as using PyTorch.
  2. Probability: You should be comfortable with all of the concepts described here.
  3. Calculus: You need to have taken a course in calculus that includes derivatives and integrals.
  4. Algorithms: You need to have taken or be taking a course in algorithms such as CS 5800 or CS 7800. If you do not have this, but would still like to take the course, please contact the instructor.
  5. Linear Algebra: You should be comfortable with all of the concepts described here.
  6. GPU: It would help to have access to a cuda-enabled GPU (with the installed drivers) for the deep RL assignments, but it's not strictly necessary for the course.

Academic Integrity

Cheating and other acts of academic dishonesty will be referred to OSCCR (office of student conduct and conflict resolution) and the College of Computer Science. See this link.

Generative AI can only be used for finding bugs or polishing English---all code and solutions should be writen up by the student.

Announcements

Our Piazza page is here.

Work Load

Required course work is:

  • Approximately weekly problem sets and/or programming assignments (40% of your grade)
  • 2 Midterms (30% of your grade) 
  • 1 Final project (30% of your grade)

Programming assignments

Many of the weekly assignments will involve programming in Python. You must be proficient with python and able to install PyTorch on your computer.

Problem sets.

We will assign approximately one assignment per week that will typically be due on Wednesday.

Final project

The final project can be on any topic related to RL. Students may work alone or in groups of tw . Many people choose to work on a project applying a method studied in the class (or a related method) to some practical problem. The amount of project work should be equivalent to approximately three assignments.

How to turn in the programming assignments

Details are on Canvas. 

Lateness Policy

Late assignments will be penalized by 10% for each day late with a maximum of two days. For example, if you turned in a perfect assignment two days late, you would receive an 80% instead of 100%.

Instruction Staff

Primary Instructor: Chris Amato ( c [dot] amato [at] neu [dot] edu )

TAs and office hours listed on Piazza