Your child's own
Reinforcement Learning tutor

MDP Foundations & the RL Problem

Learning by trial and error is framed as an agent interacting with an environment over time, choosing actions to maximize a scalar reward signal. Topics: The Reinforcement Learning Problem, Finite Markov Decision Processes, Returns & Discounting, Policies & Value Functions, Optimality & the Bellman Optimality Equations.

Multi-Armed Bandits & Exploration Foundations

Strip the problem down to a single state and ask how an agent should choose among competing actions when each only reveals its payoff by being tried. Topics: The k-Armed Bandit Problem, Action-Value Estimation, Action Selection Rules, Gradient Bandit Methods, Associative Search & Contextual Bandits.

Dynamic Programming & Planning

When the environment dynamics are fully known, value functions can be computed exactly by sweeping the Bellman equations as backups over the whole state space. Topics: Iterative Policy Evaluation, Policy Improvement, Policy & Value Iteration, Generalized Policy Iteration & Convergence.

Monte Carlo & Temporal-Difference Methods

When the environment dynamics are unknown, value functions can no longer be computed by full-width backups and must instead be learned from sampled experience. Topics: Monte Carlo Prediction, Monte Carlo Control, Off-Policy Prediction via Importance Sampling, Temporal-Difference Prediction, Temporal-Difference Control, Maximization Bias & Afterstates.

n-step Bootstrapping & Eligibility Traces

Monte Carlo and one-step temporal-difference learning are the two endpoints of a single spectrum, and this unit builds the machinery that spans between them. Topics: n-step Bootstrapping, The lambda-Return & the Forward View, Eligibility Traces & the Backward View, Online Traces in Control.

Value Function Approximation & the Deadly Triad

When the state space is too large to store one value per state, value functions must be approximated by a parameterized function whose weights generalize across states. Topics: On-Policy Prediction with Approximation, Linear Methods & Feature Construction, On-Policy Control with Approximation, Off-Policy Approximation & the Deadly Triad.

Deep Value-Based RL (DQN Family)

Once action values are represented by a neural network instead of a table, the bootstrapped Q-learning update becomes unstable: samples are correlated and the regression target shifts as the network learns. Topics: From Q-Learning to Deep Q-Networks, Overestimation & Architecture Refinements, Distributional Value RL, Multi-Step & Integrated Value RL.

Policy Gradient & Actor-Critic Methods

Instead of deriving behavior from learned values, these methods give the policy its own parameters and push them directly uphill on expected return. Topics: Policy Parameterization & the Policy Gradient Theorem, REINFORCE: Monte-Carlo Policy Gradient, Baselines & Variance Reduction, Actor-Critic Methods.

Advanced Policy Optimization

The plain policy gradient points uphill in raw parameter space, where a fixed step can change the policy's behavior wildly or barely at all, so this unit reworks the update to respect how much the policy distribution actually moves. Topics: Natural Gradient & the Fisher Metric, Trust-Region Policy Optimization, Proximal Policy Optimization, Generalized Advantage Estimation.

Continuous Control & Maximum-Entropy RL

When actions form a continuous space, the discrete max that drives value-based control becomes an intractable optimization at every step, so control instead trains a parameterized actor whose own output supplies the maximizing action. Topics: Deterministic Policy Gradients & DDPG, Overestimation & TD3 Fixes, The Maximum-Entropy RL Framework, Soft Actor-Critic.

Model-Based RL & Planning

When an agent has, or can learn, a model of how the world responds to its actions, it can plan by simulating experience rather than waiting for every transition to happen for real. Topics: Dyna & Tabular Planning, Learning Models & Their Errors, Decision-Time Planning & Tree Search, Learned-Model Deep RL.

Exploration in Deep RL

In large state spaces with sparse or deceptive rewards, randomly perturbing actions is not enough: an agent can dither forever without ever reaching the states that carry learning signal. Topics: The Deep Exploration Problem, Count-Based & Pseudo-Count Exploration, Curiosity & Prediction-Error Exploration, Posterior, Parametric-Noise & Structured Exploration.

Offline RL, Imitation & Inverse RL

Sometimes an agent must learn entirely from a fixed record of behavior, with little or no chance to try its own actions in the world. Topics: Imitation Learning & Distribution Shift, Inverse Reinforcement Learning, Adversarial Imitation Learning, Offline (Batch) Reinforcement Learning.

Frontiers: RLHF, Multi-Agent & Advanced Topics

This unit collects the modern alignment-era and advanced-theory extensions of reinforcement learning, where the reward itself is learned from human comparisons, several agents adapt at once, and the agent reasons over temporally extended behaviors. Topics: Reinforcement Learning from Human Feedback, Multi-Agent Reinforcement Learning, Hierarchical RL & Temporal Abstraction, Control as Inference & Meta-RL.