Course Schedule
The schedule is tentative and may be adjusted as the semester progresses. Readings refer to Sutton and Barto, second edition. Course-specific materials will be posted through this site and Canvas.
No class
Tutorial / Q&A
Project milestone
| Week | Date | Lecture | Material | Assignment |
|---|---|---|---|---|
| 1 | Tue, Aug 18 | Course introductionAgent–environment interaction, rewards, examples, and challenges | S&B 1.1–1.4 | — |
| 1 | Thu, Aug 20 | Multi-armed bandits IAction values, sample averages, constant step sizes, and ε-greedy | S&B 2.1–2.5 | — |
| 2 | Tue, Aug 25 | Multi-armed bandits IIOptimism, UCB, gradient bandits, nonstationarity, and regret | S&B 2.5–2.8 | — |
| 2 | Thu, Aug 27 | Markov decision processes IStates, actions, transition models, Markov property, and policies | S&B 3.1–3.4 | — |
| 3 | Tue, Sep 1 | Markov decision processes IIReturns, discounting, episodic and continuing tasks, and value functions | S&B 3.3–3.7 | Homework 1 + Coding Assignment 1 released |
| 3 | Thu, Sep 3 | Bellman equations and policy evaluation | S&B 3.5–3.6, 4.1 | — |
| 4 | Tue, Sep 8 | Policy improvement and policy iteration | S&B 4.2–4.3 | — |
| 4 | Thu, Sep 10 | Value iteration and DP | S&B 4.4–4.7 | — |
| 5 | Tue, Sep 15 | Monte Carlo prediction | S&B 5.1–5.3 | — |
| 5 | Thu, Sep 17 | Monte Carlo control and off-policy learning | S&B 5.4–5.7 | — |
| 6 | Tue, Sep 22 | Temporal-difference predictionTD(0), bootstrapping, and MC–TD comparison | S&B 6.1–6.3 | — |
| 6 | Thu, Sep 24 | TA-led coding tutorial: tabular TD controlImplement SARSA and Q-learning, run controlled experiments, and compare their learning behavior | S&B 6.4–6.7 | Homework 1 + Coding Assignment 1 DUE |
| 7 | Tue, Sep 29 | No class — IROS travelOptional guided review | — | Homework 2 + Coding Assignment 2 released |
| 7 | Thu, Oct 1 | No class — IROS travel | — | — |
| 8 | Tue, Oct 6 | No class — Fall Break | — | — |
| 10 | Tue, Oct 8 | Homework and coding assignment Q&A / review sessionTroubleshoot Homework and Coding Assignment 1-2, review common issues | — | Bring your questions |
| 9 | Tue, Oct 13 | Combined project (building a complete RL system) — midterm milestoneProject integration | Project specification | Bring a laptop |
| 9 | Thu, Oct 15 | Combined project (building a complete RL system) — midterm milestoneSystem demonstrations, testing, feedback, and milestone review | Project specification | Bring a laptop |
| 8 | Thu, Oct 20 | n-step TD | S&B 7.1–7.3; selected Ch. 12 | Final project phase begins: train an RL agent in simulation |
| 10 | Thu, Oct 22 | Planning and learning IDyna, direct and indirect learning, and simulated experience | S&B 8.1–8.3 | Homework 2 + Coding Assignment 2 DUE |
| 11 | Tue, Oct 27 | Planning and learning IIPrioritized sweeping, trajectories, and planning at decision time | S&B 8.4–8.11 | Homework 3 + Coding Assignment 3 released |
| 11 | Thu, Oct 29 | Value function approximation ILinear features, objectives, and semi-gradient prediction | S&B 9.1–9.5 | — |
| 12 | Tue, Nov 3 | No class — Election Day | — | — |
| 12 | Thu, Nov 5 | Value function approximation IIControl, off-policy instability, and the deadly triad | S&B 10.1, 11.1 | — |
| 13 | Tue, Nov 10 | Policy gradients IParameterized policies, score functions, and REINFORCE | S&B 13.1–13.3 | Final project progress check (submit a short summary report) |
| 13 | Thu, Nov 12 | Policy gradients IIBaselines, actor–critic, variance, and continuing tasks | S&B 13.4–13.5 | — |
| 14 | Tue, Nov 17 | Deep reinforcement learningDQN, experience replay, target networks, representation learning, and training stability | — | — |
| 14 | Thu, Nov 19 | Reward design and learning from preferencesReward hypothesis, reward shaping, inverse reinforcement learning, and preference-based feedback | — | Homework 3 + Coding Assignment 3 DUE |
| 15 | Tue, Nov 24 | RL beyond the Markov propertyPartial observability, state representation, memory, belief states, and recurrent agents | — | — |
| 15 | Thu, Nov 26 | No class — Thanksgiving Recess | — | — |
| 16 | Tue, Dec 1 | Final project poster and demo sessionsPresent the completed system, trained RL agent, experimental results, and lessons learned | Poster and demonstration guidelines | Project poster and live demo |
UTK calendar notes: Full-session classes end Tuesday, December 1; Study Day is Wednesday, December 2; final exams run December 3–9. Final project deadlines will be confirmed in the class.