Sutton, Richard S., <br/><br/>
Reinforcement learning :  an introduction / 
Richard S. Sutton and Andrew G. Barto. 
 - Second edition. 
 - xxii, 526 pages :  illustrations (some color) ;  24 cm. 
 - Adaptive computation and machine learning series .
<br/><br/> The significantly expanded and updated new edition of a widely used text on reinforcement learning, one of the most active research areas in artificial intelligence.<br/><br/>Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives while interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the field's key ideas and algorithms. This second edition has been significantly expanded and updated, presenting new topics and updating coverage of other topics.<br/><br/>Like the first edition, this second edition focuses on core online learning algorithms, with the more mathematical material set off in shaded boxes. Part I covers as much of reinforcement learning as possible without going beyond the tabular case for which exact solutions can be found. Many algorithms presented in this part are new to the second edition, including UCB, Expected Sarsa, and Double Learning. Part II extends these ideas to function approximation, with new sections on such topics as artificial neural networks and the Fourier basis, and offers expanded treatment of off-policy learning and policy-gradient methods. Part III has new chapters on reinforcement learning's relationships to psychology and neuroscience, as well as an updated case-studies chapter including AlphaGo and AlphaGo Zero, Atari game playing, and IBM Watson's wagering strategy. The final chapter discusses the future societal impacts of reinforcement learning. 
<br/><br/>Includes bibliographical references (pages 481-518) and index. 
<br/><br/>Contents<br/>Preface to the Second Edition xiii<br/>Preface to the First Edition xvii<br/>Summary of Notation xix<br/>1 Introduction 1<br/>1.1 Reinforcement Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1<br/>1.2 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4<br/>1.3 Elements of Reinforcement Learning . . . . . . . . . . . . . . . . . . . . . 6<br/>1.4 Limitations and Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7<br/>1.5 An Extended Example: Tic-Tac-Toe . . . . . . . . . . . . . . . . . . . . . 8<br/>1.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13<br/>1.7 Early History of Reinforcement Learning . . . . . . . . . . . . . . . . . . . 13<br/>I Tabular Solution Methods 23<br/>2 Multi-armed Bandits 25<br/>2.1 A k -armed Bandit Problem . . . . . . . . . . . . . . . . . . . . . . . . . . 25<br/>2.2 Action-value Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27<br/>2.3 The 10-armed Testbed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28<br/>2.4 Incremental Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 30<br/>2.5 Tracking a Nonstationary Problem . . . . . . . . . . . . . . . . . . . . . . 32<br/>2.6 Optimistic Initial Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34<br/>2.7 Upper-Confidence-Bound Action Selection . . . . . . . . . . . . . . . . . . 35<br/>2.8 Gradient Bandit Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . 37<br/>2.9 Associative Search (Contextual Bandits) . . . . . . . . . . . . . . . . . . . 41<br/>2.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42<br/>viii Contents<br/>3 Finite Markov Decision Processes 47<br/>3.1 The Agent–Environment Interface . . . . . . . . . . . . . . . . . . . . . . 47<br/>3.2 Goals and Rewards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53<br/>3.3 Returns and Episodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54<br/>3.4 Unified Notation for Episodic and Continuing Tasks . . . . . . . . . . . . 57<br/>3.5 Policies and Value Functions . . . . . . . . . . . . . . . . . . . . . . . . . 58<br/>3.6 Optimal Policies and Optimal Value Functions . . . . . . . . . . . . . . . 62<br/>3.7 Optimality and Approximation . . . . . . . . . . . . . . . . . . . . . . . . 67<br/>3.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68<br/>4 Dynamic Programming 73<br/>4.1 Policy Evaluation (Prediction) . . . . . . . . . . . . . . . . . . . . . . . . 74<br/>4.2 Policy Improvement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76<br/>4.3 Policy Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80<br/>4.4 Value Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82<br/>4.5 Asynchronous Dynamic Programming . . . . . . . . . . . . . . . . . . . . 85<br/>4.6 Generalized Policy Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . 86<br/>4.7 Efficiency of Dynamic Programming . . . . . . . . . . . . . . . . . . . . . 87<br/>4.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88<br/>5 Monte Carlo Methods 91<br/>5.1 Monte Carlo Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92<br/>5.2 Monte Carlo Estimation of Action Values . . . . . . . . . . . . . . . . . . 96<br/>5.3 Monte Carlo Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97<br/>5.4 Monte Carlo Control without Exploring Starts . . . . . . . . . . . . . . . 100<br/>5.5 Off-policy Prediction via Importance Sampling . . . . . . . . . . . . . . . 103<br/>5.6 Incremental Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 109<br/>5.7 Off-policy Monte Carlo Control . . . . . . . . . . . . . . . . . . . . . . . . 110<br/>5.8 *Discounting-aware Importance Sampling . . . . . . . . . . . . . . . . . . 112<br/>5.9 *Per-decision Importance Sampling . . . . . . . . . . . . . . . . . . . . . . 114<br/>5.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115<br/>6 Temporal-Difference Learning 119<br/>6.1 TD Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119<br/>6.2 Advantages of TD Prediction Methods . . . . . . . . . . . . . . . . . . . . 124<br/>6.3 Optimality of TD(0) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126<br/>6.4 Sarsa: On-policy TD Control . . . . . . . . . . . . . . . . . . . . . . . . . 129<br/>6.5 Q-learning: Off-policy TD Control . . . . . . . . . . . . . . . . . . . . . . 131<br/>6.6 Expected Sarsa . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133<br/>6.7 Maximization Bias and Double Learning . . . . . . . . . . . . . . . . . . . 134<br/>6.8 Games, Afterstates, and Other Special Cases . . . . . . . . . . . . . . . . 136<br/>6.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138<br/>Contents ix<br/>7 n-step Bootstrapping 141<br/>7.1 n-step TD Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142<br/>7.2 n-step Sarsa . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145<br/>7.3 n-step Off-policy Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . 148<br/>7.4 *Per-decision Methods with Control Variates . . . . . . . . . . . . . . . . 150<br/>7.5 Off-policy Learning Without Importance Sampling:<br/>The n-step Tree Backup Algorithm . . . . . . . . . . . . . . . . . . . . . . 152<br/>7.6 *A Unifying Algorithm: n-step Q(σ) . . . . . . . . . . . . . . . . . . . . . 154<br/>7.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157<br/>8 Planning and Learning with Tabular Methods 159<br/>8.1 Models and Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159<br/>8.2 Dyna: Integrated Planning, Acting, and Learning . . . . . . . . . . . . . . 161<br/>8.3 When the Model Is Wrong . . . . . . . . . . . . . . . . . . . . . . . . . . . 166<br/>8.4 Prioritized Sweeping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168<br/>8.5 Expected vs. Sample Updates . . . . . . . . . . . . . . . . . . . . . . . . . 172<br/>8.6 Trajectory Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174<br/>8.7 Real-time Dynamic Programming . . . . . . . . . . . . . . . . . . . . . . . 177<br/>8.8 Planning at Decision Time . . . . . . . . . . . . . . . . . . . . . . . . . . . 180<br/>8.9 Heuristic Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181<br/>8.10 Rollout Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183<br/>8.11 Monte Carlo Tree Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185<br/>8.12 Summary of the Chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188<br/>8.13 Summary of Part I: Dimensions . . . . . . . . . . . . . . . . . . . . . . . . 189<br/>II Approximate Solution Methods 195<br/>9 On-policy Prediction with Approximation 197<br/>9.1 Value-function Approximation . . . . . . . . . . . . . . . . . . . . . . . . . 198<br/>9.2 The Prediction Objective (VE) . . . . . . . . . . . . . . . . . . . . . . . . 199<br/>9.3 Stochastic-gradient and Semi-gradient Methods . . . . . . . . . . . . . . . 200<br/>9.4 Linear Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204<br/>9.5 Feature Construction for Linear Methods . . . . . . . . . . . . . . . . . . 210<br/>9.5.1 Polynomials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210<br/>9.5.2 Fourier Basis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211<br/>9.5.3 Coarse Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215<br/>9.5.4 Tile Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217<br/>9.5.5 Radial Basis Functions . . . . . . . . . . . . . . . . . . . . . . . . . 221<br/>9.6 Selecting Step-Size Parameters Manually . . . . . . . . . . . . . . . . . . . 222<br/>9.7 Nonlinear Function Approximation: Artificial Neural Networks . . . . . . 223<br/>9.8 Least-Squares TD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228<br/>x Contents<br/>9.9 Memory-based Function Approximation . . . . . . . . . . . . . . . . . . . 230<br/>9.10 Kernel-based Function Approximation . . . . . . . . . . . . . . . . . . . . 232<br/>9.11 Looking Deeper at On-policy Learning: Interest and Emphasis . . . . . . 234<br/>9.12 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236<br/>10 On-policy Control with Approximation 243<br/>10.1 Episodic Semi-gradient Control . . . . . . . . . . . . . . . . . . . . . . . . 243<br/>10.2 Semi-gradient n-step Sarsa . . . . . . . . . . . . . . . . . . . . . . . . . . 247<br/>10.3 Average Reward: A New Problem Setting for Continuing Tasks . . . . . . 249<br/>10.4 Deprecating the Discounted Setting . . . . . . . . . . . . . . . . . . . . . . 253<br/>10.5 Differential Semi-gradient n-step Sarsa . . . . . . . . . . . . . . . . . . . . 255<br/>10.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256<br/>11 *Off-policy Methods with Approximation 257<br/>11.1 Semi-gradient Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258<br/>11.2 Examples of Off-policy Divergence . . . . . . . . . . . . . . . . . . . . . . 260<br/>11.3 The Deadly Triad . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264<br/>11.4 Linear Value-function Geometry . . . . . . . . . . . . . . . . . . . . . . . 266<br/>11.5 Gradient Descent in the Bellman Error . . . . . . . . . . . . . . . . . . . . 269<br/>11.6 The Bellman Error is Not Learnable . . . . . . . . . . . . . . . . . . . . . 274<br/>11.7 Gradient-TD Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278<br/>11.8 Emphatic-TD Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281<br/>11.9 Reducing Variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283<br/>11.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284<br/>12 Eligibility Traces 287<br/>12.1 The λ-return . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288<br/>12.2 TD(λ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292<br/>12.3 n-step Truncated λ-return Methods . . . . . . . . . . . . . . . . . . . . . . 295<br/>12.4 Redoing Updates: Online λ-return Algorithm . . . . . . . . . . . . . . . . 297<br/>12.5 True Online TD(λ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299<br/>12.6 *Dutch Traces in Monte Carlo Learning . . . . . . . . . . . . . . . . . . . 301<br/>12.7 Sarsa(λ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303<br/>12.8 Variable λ and γ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307<br/>12.9 Off-policy Traces with Control Variates . . . . . . . . . . . . . . . . . . . 309<br/>12.10 Watkins’s Q(λ) to Tree-Backup(λ) . . . . . . . . . . . . . . . . . . . . . . 312<br/>12.11 Stable Off-policy Methods with Traces . . . . . . . . . . . . . . . . . . . 314<br/>12.12 Implementation Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316<br/>12.13 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317<br/>Contents xi<br/>13 Policy Gradient Methods 321<br/>13.1 Policy Approximation and its Advantages . . . . . . . . . . . . . . . . . . 322<br/>13.2 The Policy Gradient Theorem . . . . . . . . . . . . . . . . . . . . . . . . . 324<br/>13.3 REINFORCE: Monte Carlo Policy Gradient . . . . . . . . . . . . . . . . . 326<br/>13.4 REINFORCE with Baseline . . . . . . . . . . . . . . . . . . . . . . . . . . 329<br/>13.5 Actor–Critic Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331<br/>13.6 Policy Gradient for Continuing Problems . . . . . . . . . . . . . . . . . . 333<br/>13.7 Policy Parameterization for Continuous Actions . . . . . . . . . . . . . . . 335<br/>13.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337<br/>III Looking Deeper 339<br/>14 Psychology 341<br/>14.1 Prediction and Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342<br/>14.2 Classical Conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343<br/>14.2.1 Blocking and Higher-order Conditioning . . . . . . . . . . . . . . . 345<br/>14.2.2 The Rescorla–Wagner Model . . . . . . . . . . . . . . . . . . . . . 346<br/>14.2.3 The TD Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349<br/>14.2.4 TD Model Simulations . . . . . . . . . . . . . . . . . . . . . . . . . 350<br/>14.3 Instrumental Conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . . 357<br/>14.4 Delayed Reinforcement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361<br/>14.5 Cognitive Maps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363<br/>14.6 Habitual and Goal-directed Behavior . . . . . . . . . . . . . . . . . . . . . 364<br/>14.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368<br/>15 Neuroscience 377<br/>15.1 Neuroscience Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 378<br/>15.2 Reward Signals, Reinforcement Signals, Values, and Prediction Errors . . 380<br/>15.3 The Reward Prediction Error Hypothesis . . . . . . . . . . . . . . . . . . 381<br/>15.4 Dopamine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383<br/>15.5 Experimental Support for the Reward Prediction Error Hypothesis . . . . 387<br/>15.6 TD Error/Dopamine Correspondence . . . . . . . . . . . . . . . . . . . . . 390<br/>15.7 Neural Actor–Critic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395<br/>15.8 Actor and Critic Learning Rules . . . . . . . . . . . . . . . . . . . . . . . 398<br/>15.9 Hedonistic Neurons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402<br/>15.10 Collective Reinforcement Learning . . . . . . . . . . . . . . . . . . . . . . 404<br/>15.11 Model-based Methods in the Brain . . . . . . . . . . . . . . . . . . . . . . 407<br/>15.12 Addiction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409<br/>15.13 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410<br/>xii Contents<br/>16 Applications and Case Studies 421<br/>16.1 TD-Gammon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 421<br/>16.2 Samuel’s Checkers Player . . . . . . . . . . . . . . . . . . . . . . . . . . . 426<br/>16.3 Watson’s Daily-Double Wagering . . . . . . . . . . . . . . . . . . . . . . . 429<br/>16.4 Optimizing Memory Control . . . . . . . . . . . . . . . . . . . . . . . . . . 432<br/>16.5 Human-level Video Game Play . . . . . . . . . . . . . . . . . . . . . . . . 436<br/>16.6 Mastering the Game of Go . . . . . . . . . . . . . . . . . . . . . . . . . . . 441<br/>16.6.1 AlphaGo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444<br/>16.6.2 AlphaGo Zero . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 447<br/>16.7 Personalized Web Services . . . . . . . . . . . . . . . . . . . . . . . . . . . 450<br/>16.8 Thermal Soaring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453<br/>17 Frontiers 459<br/>17.1 General Value Functions and Auxiliary Tasks . . . . . . . . . . . . . . . . 459<br/>17.2 Temporal Abstraction via Options . . . . . . . . . . . . . . . . . . . . . . 461<br/>17.3 Observations and State . . . . . . . . . . . . . . . . . . . . . . . . . . . . 464<br/>17.4 Designing Reward Signals . . . . . . . . . . . . . . . . . . . . . . . . . . . 469<br/>17.5 Remaining Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 472<br/>17.6 Experimental Support for the Reward Prediction Error Hypothesis . . . . 475<br/>References 481<br/>Index 519<br/> 
<br/><br/> "Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives while interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the field's key ideas and algorithms."-- 
<br/><br/><label>ISBN: </label>9780262039246 (hardcover : alk. paper) 
<br/><br/><label>LCCN: </label>2018023826 
<br/><br/><label>Subjects--Topical Terms: </label><br/>Reinforcement learning.
<br/><br/><label>LC Class. No.: </label>Q325.6  / .R45 2018 
<br/><br/><label>Dewey Class. No.: </label>006.31 SUT