Overview of this BookChapter 1 Basic Concepts1.1 A grid world example1.2 State and action1.3 State transition1.4 Policy1.5 Reward1.6 Trajectories, returns, and episodes1.7 Markov decision processes1.8 Summary1.9 Q&AChapter 2 State Values and the Bellman Equation2.1 Motivating example 1: Why are returns important?2.2 Motivating example 2: How to calculate returns?2.3 State values2.4 The Bellman equation2.5 Examples for illustrating the Bellman equation2.6 Matrix-vector form of the Bellman equation2.7 Solving state values from the Bellman equation2.7.1 Closed-form solution2.7.2 Iterative solution2.7.3 Illustrative examples2.8 From state value to action value2.8.1 Illustrative examples2.8.2 The Bellman equation in terms of action values2.9 Summary2.10 Q&AChapter 3 Optimal State Values and the Bellman Optimality Equation3.1 Motivating example: How to improve policies?3.2 Optimal state values and optimal policies3.3 The Bellman optimality equation3.3.1 Maximization of the right-hand side of the BOE3.3.2 Matrix-vector form of the BOE3.3.3 Contraction mapping theorem3.3.4 Contraction property of the right-hand side of the BOE3.4 Solving an optimal policy from the BOE3.5 Factors that influence optimal policies3.6 Summary3.7 Q&AChapter 4 Value Iteration and Policy Iteration4.1 Value iteration4.1.1 Elementwise form and implementation4.1.2 Illustrative examples4.2 Policy iteration4.2.1 Algorithm analysis4.2.2 Elementwise form and implementation4.2.3 Illustrative examples4.3 Truncated policy iteration4.3.1 Comparing value iteration and policy iteration4.3.2 Truncated policy iteration algorithm4.4 Summary4.5 Q&AChapter 5 Monte Carlo Methods5.1 Motivating example: Mean estimation5.2 MC Basic: The simplest MC-based algorithm5.2.1 Converting policy iteration to be model-free5.2.2 The MC Basic algorithm5.2.3 Illustrative examples5.3 MC Exploring Starts5.3.1 Utilizing samples more efficiently5.3.2 Updating policies more efficiently5.3.3 Algorithm description5.4 MC ∈-Greedy: Learning without exploring starts5.4.1 ∈-greedy policies5.4.2 Algorithm description5.4.3 Illustrative examples5.5 Exploration and exploitation of ∈-greedy policies5.6 Summary5.7 Q&AChapter 6 Stochastic Approximation6.1 Motivating example: Mean estimation6.2 Robbins-Monro algorithm6.2.1 Convergence properties6.2.2 Application to mean estimation6.3 Dvoretzky's convergence theorem6.3.1 Proof of Dvoretzky's theorem6.3.2 Application to mean estimation6.3.3 Application to the Robbins-Monro theorem6.3.4 An extension of Dvoretzky's theorem6.4 Stochastic gradient descent6.4.1 Application to mean estimation6.4.2 Convergence pattern of SGD6.4.3 A deterministic formulation of SGD6.4.4 BGD, SGD, and mini-batch GD6.4.5 Convergence of SGD6.5 Summary6.6 Q&AChapter 7 Temporal-Difference Methods7.1 TD learning of state values7.1.1 Algorithm description7.1.2 Property analysis7.1.3 Convergence analysis7.2 TD learning of action values: Sarsa7.2.1 Algorithm description7.2.2 Optimal policy learning via Sarsa7.3 TD learning of action values: n-step Sarsa7.4 TD learning of optimal action values: Q-learning7.4.1 Algorithm description7.4.2 Off-policy vs. on-policy7.4.3 Implementation7.4.4 Illustrative examples7.5 A unifed viewpoint7.6 Summary7.7 Q&AChapter 8 Value Function Approximation8.1 Value representation: From table to function8.2 TD learning of state values with function approximation8.2.1 O