Course Schedule
The schedule is tentative and may be adjusted as the semester progresses. Readings refer to Sutton and Barto, second edition. Course-specific materials will be posted through this site and Canvas.
No class
Tutorial / Q&A
Project milestone
| Week | Date | Lecture | Material | Assignment |
|---|---|---|---|---|
| 1 | Tue, Aug 18 | Course introductionAgent-environment interaction, rewards, examples, and challenges | S&B 1.1–1.4 | — |
| 1 | Thu, Aug 20 | Multi-armed banditsAction-value methods, exploration versus exploitation, and nonstationary problems | S&B 2.1–2.5 | — |
| 2 | Tue, Aug 25 | Markov Decision Processes IStates, actions, transition models, Markov property, and policies | S&B 3.1–3.2 | — |
| 2 | Thu, Aug 27 | Markov Decision Processes IIGoals and rewards, returns, episodic and continuing tasks, and unified notation | S&B 3.3–3.4 | — |
| 3 | Tue, Sep 1 | Dynamic Programming I: Value Functions and Bellman EquationsState- and action-value functions, Bellman expectation equations, and optimal value functions | S&B 3.5–3.7 | Homework 1 + Coding Assignment 1 released |
| 3 | Thu, Sep 3 | Dynamic Programming II: Policy Evaluation and Policy IterationIterative policy evaluation, policy improvement, and policy iteration | S&B 4.1–4.3 | — |
| 4 | Tue, Sep 8 | Dynamic Programming III: Value IterationBellman optimality equations, value iteration, generalized policy iteration, examples | S&B 4.4–4.7 | — |
| 4 | Thu, Sep 10 | Monte Carlo PredictionModel-free value estimation, first-visit and every-visit methods, and sample-average returns | S&B 5.1–5.2 | — |
| 5 | Tue, Sep 15 | Monte Carlo ControlExploring starts, on-policy control, and soft policies | S&B 5.3–5.4 | — |
| 5 | Thu, Sep 17 | Off-policy Monte Carlo MethodsImportance sampling, off-policy prediction, and off-policy control | S&B 5.5–5.7 | — |
| 6 | Tue, Sep 22 | Temporal-difference PredictionTD(0), bootstrapping, and MC–TD comparison | S&B 6.1–6.3 | — |
| 6 | Thu, Sep 24 | TA-led coding tutorial: tabular TD controlImplement SARSA and Q-learning, run controlled experiments, and compare their learning behavior | S&B 6.4–6.7 | Homework 1 + Coding Assignment 1 DUE |
| 7 | Tue, Sep 29 | No class — IROS travelOptional guided review | — | Homework 2 + Coding Assignment 2 released |
| 7 | Thu, Oct 1 | No class — IROS travel | — | — |
| 8 | Tue, Oct 6 | No class — Fall Break | — | — |
| 8 | Thu, Oct 8 | Homework and coding assignment Q&A / review sessionTroubleshoot Homework and Coding Assignment 1-2, review common issues | — | Bring your questions |
| 9 | Tue, Oct 13 | Combined project (building a complete RL system) — midterm milestoneProject integration | Project specification | Bring a laptop |
| 9 | Thu, Oct 15 | Combined project (building a complete RL system) — midterm milestoneSystem demonstrations, testing, feedback, and milestone review | Project specification | Bring a laptop |
| 10 | Tue, Oct 20 | n-step Temporal-difference Methodsn-step prediction, n-step SARSA, and off-policy n-step learning | S&B 7.1–7.6 | Final project phase begins: train an RL agent in simulation |
| 10 | Thu, Oct 22 | Planning and learning IDyna, direct and indirect learning, and simulated experience | S&B 8.1–8.3 | Homework 2 + Coding Assignment 2 DUE |
| 11 | Tue, Oct 27 | Planning and learning IIPrioritized sweeping, trajectories, and planning at decision time | S&B 8.4–8.11 | Homework 3 + Coding Assignment 3 released |
| 11 | Thu, Oct 29 | Value function approximation ILinear features, objectives, and semi-gradient prediction | S&B 9.1–9.5 | — |
| 12 | Tue, Nov 3 | No class — Election Day | — | — |
| 12 | Thu, Nov 5 | Value function approximation IIControl, off-policy instability, and the deadly triad | S&B 10.1–10.2, 11.1–11.3 | — |
| 13 | Tue, Nov 10 | Policy gradients IParameterized policies, score functions, and REINFORCE | S&B 13.1–13.3 | Final project progress check (submit a short summary report) |
| 13 | Thu, Nov 12 | Policy gradients IIBaselines, actor–critic, variance, and continuing tasks | S&B 13.4–13.6 | — |
| 14 | Tue, Nov 17 | Deep reinforcement learningDQN, experience replay, target networks, representation learning, and training stability | Mnih et al. (2015); selected notes | — |
| 14 | Thu, Nov 19 | Reward design and learning from preferencesReward hypothesis, reward shaping, inverse reinforcement learning, and preference-based feedback | Selected readings and lecture notes | Homework 3 + Coding Assignment 3 DUE |
| 15 | Tue, Nov 24 | RL beyond the Markov propertyPartial observability, state representation, memory, belief states, and recurrent agents | Selected notes on POMDPs and recurrent agents | — |
| 15 | Thu, Nov 26 | No class — Thanksgiving Recess | — | — |
| 16 | Tue, Dec 1 | Final project poster and demo sessionsPresent the completed system, trained RL agent, experimental results, and lessons learned | Poster and demonstration guidelines | Project poster and live demo |
UTK calendar notes: Full-session classes end Tuesday, December 1; Study Day is Wednesday, December 2; final exams run December 3–9. Final project deadlines will be confirmed in the class.