PREFACE xix
CONTRIBUTORS xxiii
PART I FEEDBACK CONTROL USING RL AND ADP
1. Reinforcement Learning and Approximate Dynamic Programming (RLADP)-Foundations, Common Misconceptions, and the Challenges Ahead 3
Paul J. Werbos
1.1 Introduction 3
1.2 What is RLADP? 4
1.3 Some Basic Challenges in Implementing ADP 14
2. Stable Adaptive Neural Control of Partially Observable Dynamic Systems 31
J. Nate Knight and Charles W. Anderson
2.1 Introduction 31
2.2 Background 32
2.3 Stability Bias 35
2.4 Example Application 38
3. Optimal Control of Unknown Nonlinear Discrete-Time Systems Using the Iterative Globalized Dual Heuristic Programming Algorithm 52
Derong Liu and Ding Wang
3.1 Background Material 53
3.2 Neuro-Optimal Control Scheme Based on the Iterative ADP Algorithm 55
3.3 Generalization 67
3.4 Simulation Studies 68
3.5 Summary 74
4. Learning and Optimization in Hierarchical Adaptive Critic Design 78
Haibo He, Zhen Ni, and Dongbin Zhao
4.1 Introduction 78
4.2 Hierarchical ADP Architecture with Multiple-Goal Representation 80
4.3 Case Study: The Ball-and-Beam System 87
4.4 Conclusions and Future Work 94
5. Single Network Adaptive Critics Networks-Development, Analysis, and Applications 98
Jie Ding, Ali Heydari, and S.N. Balakrishnan
5.1 Introduction 98
5.2 Approximate Dynamic Programing 100
5.3 SNAC 102
5.4 J-SNAC 104
5.5 Finite-SNAC 108
5.6 Conclusions 116
6. Linearly Solvable Optimal Control 119
K. Dvijotham and E. Todorov
6.1 Introduction 119
6.2 Linearly Solvable Optimal Control Problems 123
6.3 Extension to Risk-Sensitive Control and Game Theory 130
6.4 Properties and Algorithms 134
6.5 Conclusions and Future Work 139
7. Approximating Optimal Control with Value Gradient Learning 142
Michael Fairbank, Danil Prokhorov, and Eduardo Alonso
7.1 Introduction 142
7.2 Value Gradient Learning and BPTT Algorithms 144
7.3 A Convergence Proof for VGL(1) for Control with Function Approximation 148
7.4 Vertical Lander Experiment 154
7.5 Conclusions 159
8. A Constrained Backpropagation Approach to Function Approximation and Approximate Dynamic Programming 162
Silvia Ferrari, Keith Rudd, and Gianluca Di Muro
8.1 Background 163
8.2 Constrained Backpropagation (CPROP) Approach 163
8.3 Solution of Partial Differential Equations in Nonstationary Environments 170
8.4 Preserving Prior Knowledge in Exploratory Adaptive Critic Designs 174
8.5 Summary 179
9. Toward Design of Nonlinear ADP Learning Controllers with Performance Assurance 182
Jennie Si, Lei Yang, Chao Lu, Kostas S. Tsakalis, and Armando A. Rodriguez
9.1 Introduction 183
9.2 Direct Heuristic Dynamic Programming 184
9.3 A Control Theoretic View on the Direct HDP 186
9.4 Direct HDP Design with Improved Performance Case 1-Design Guided by a Priori LQR Information 193
9.5 Direct HDP Design with Improved Performance Case 2-Direct HDP for Coorindated Damping Control of Low-Frequency Oscillation 198
9.6 Summary 201
10. Reinforcement Learning Control with Time-Dependent Agent Dynamics 203
Kenton Kirkpatrick and John Valasek
10.1 Introduction 203
10.2 Q-Learning 205
10.3 Sampled Data Q-Learning 209
10.4 System Dynamics Approximation 213
10.5 Closing Remarks 218
11. Online Optimal Control of Nonaffine Nonlinear Discrete-Time Systems without Using Value and Policy Iterations 221
Hassan Zargarzadeh, Qinmin Yang, and S. Jagannathan
11.1 Introduction 221
11.2 Background 224
11.3 Reinforcement Learning Based Control 225
11.4 Time-Based Adaptive Dynamic Programming-Based Optimal Control 234
11.5 Simulation Result 247
12. An Actor-Critic-Identifier Architecture for Adaptive Approximate Optimal Control 258
S. Bhasin, R. Kamalapurkar, M. Johnson, K.G. Vamvoudakis, F.L. Lewis, and W.E. Dixon
12.1 Introduction 259
12.2 Actor-Critic-Identifier Architecture for HJB Approximation 260
12.3 Actor-Critic Design 263
12.4 Identifier Design 264
12.5 Convergence and Stability Analysis 270
12.6 Simulation 274
12.7 Conclusion 275
13. Robust Adaptive Dynamic Programming 281
Yu Jiang and Zhong-Ping Jiang
13.1 Introduction 281
13.2 Optimality Versus Robustness 283
13.3 Robust-ADP Design for Disturbance Attenuation 288
13.4 Robust-ADP for Partial-State Feedback Control 292
13.5 Applications 296
13.6 Summary 300
PART II LEARNING AND CONTROL IN MULTIAGENT GAMES
14. Hybrid Learning in Stochastic Games and Its Application in Network Security 305
Quanyan Zhu, Hamidou Tembine, and Tamer Basar
14.1 Introduction 305
14.2 Two-Person Game 308
14.3 Learning in NZSGs 310
14.4 Main Results 314
14.5 Security Application 322
14.6 Conclusions and Future Works 326
15. Integral Reinforcement Learning for Online Computation of Nash Strategies of Nonzero-Sum Differential Games 330
Draguna Vrabie and F.L. Lewis
15.1 Introduction 331
15.2 Two-Player Games and Integral Reinforcement Learning 333
15.3 Continuous-Time Value Iteration to Solve the Riccati Equation 337
15.4 Online Algorithm to Solve Nonzero-Sum Games 339
15.5 Analysis of the Online Learning Algorithm for NZS Games 342
15.6 Simulation Result for the Online Game Algorithm 345
15.7 Conclusion 347
16. Online Learning Algorithms for Optimal Control and Dynamic Games 350
Kyriakos G. Vamvoudakis and Frank L. Lewis
16.1 Introduction 350
16.2 Optimal Control and the Continuous Time Hamilton-Jacobi-Bellman Equation 352
16.3 Online Solution of Nonlinear Two-Player Zero-Sum Games and Hamilton-Jacobi-Isaacs Equation 360
16.4 Online Solution of Nonlinear Nonzero-Sum Games and Coupled Hamilton-Jacobi Equations 366
PART III FOUNDATIONS IN MDP AND RL
17. Lambda-Policy Iteration: A Review and a New Implementation 381
Dimitri P. Bertsekas
17.1 Introduction 381
17.2 Lambda-Policy Iteration without Cost Function Approximation 386
17.3 Approximate Policy Evaluation Using Projected Equations 388
17.4 Lambda-Policy Iteration with Cost Function Approximation 395
17.5 Conclusions 406
18. Optimal Learning and Approximate Dynamic Programming 410
Warren B. Powell and Ilya O. Ryzhov
18.1 Introduction 410
18.2 Modeling 411
18.3 The Four Classes of Policies 412
18.4 Basic Learning Policies for Policy Search 416
18.5 Optimal Learning Policies for Policy Search 421
18.6 Learning with a Physical State 427
19. An Introduction to Event-Based Optimization: Theory and Applications 432
Xi-Ren Cao, Yanjia Zhao, Qing-Shan Jia, and Qianchuan Zhao
19.1 Introduction 432
19.2 Literature Review 433
19.3 Problem Formulation 434
19.4 Policy Iteration for EBO 435
19.5 Example: Material Handling Problem 44119.6 Conclusions 448
20. Bounds for Markov Decision Processes 452
Vijay V. Desai, Vivek F. Farias, and Ciamac C. Moallemi
20.1 Introduction 452
20.2 Problem Formulation 455
20.3 The Linear Programming Approach 456
20.4 The Martingale Duality Approach 458
20.5 The Pathwise Optimization Method 461
20.6 Applications 463
20.7 Conclusion 470
21. Approximate Dynamic Programming and Backpropagation on Timescales 474
John Seiffertt and Donald Wunsch
21.1 Introduction: Timescales Fundamentals 474
21.2 Dynamic Programming 479
21.3 Backpropagation 485
21.4 Conclusions 492
22. A Survey of Optimistic Planning in Markov Decision Processes 494
Lucian Busoniu, Remi Munos, and Robert Babuska
22.1 Introduction 494
22.2 Optimistic Online Optimization 497
22.3 Optimistic Planning Algorithms 500
22.4 Related Planning Algorithms 509
22.5 Numerical Example 510
23. Adaptive Feature Pursuit: Online Adaptation of Features in Reinforcement Learning 517
Shalabh Bhatnagar, Vivek S. Borkar, and L.A. Prashanth
23.1 Introduction 517
23.2 The Framework 520
23.3 The Feature Adaptation Scheme 522
23.4 Convergence Analysis 525
23.5 Application to Traffic Signal Control 527
23.6 Conclusions 532
24. Feature Selection for Neuro-Dynamic Programming 535
Dayu Huang, W. Chen, P. Mehta, S. Meyn, and A. Surana
24.1 Introduction 535
24.2 Optimality Equations 536
24.3 Neuro-Dynamic Algorithms 542
24.4 Fluid Models 551
24.5 Diffusion Models 554
24.6 Mean Field Games 556
24.7 Conclusions 557
25. Approximate Dynamic Programming for Optimizing Oil Production 560
Zheng Wen, Louis J. Durlofsky, Benjamin Van Roy, and Khalid Aziz
25.1 Introduction 560
25.2 Petroleum Reservoir Production Optimization Problem 562
25.3 Review of Dynamic Programming and Approximate Dynamic Programming 564
25.4 Approximate Dynamic Programming Algorithm for Reservoir Production Optimization 566
25.5 Simulation Results 573
25.6 Concluding Remarks 578
23.6 Conclusions 532
24. Feature Selection for Neuro-Dynamic Programming 535
Dayu Huang, W. Chen, P. Mehta, S. Meyn, and A. Surana
24.1 Introduction 535
24.2 Optimality Equations 536
24.3 Neuro-Dynamic Algorithms 542
24.4 Fluid Models 551
24.5 Diffusion Models 554
24.6 Mean Field Games 556
24.7 Conclusions 557
25. Approximate Dynamic Programming for Optimizing Oil Production 560
Zheng Wen, Louis J. Durlofsky, Benjamin Van Roy, and Khalid Aziz
25.1 Introduction 560
25.2 Petroleum Reservoir Production Optimization Problem 562
25.3 Review of Dynamic Programming and Approximate Dynamic Programming 564
25.4 Approximate Dynamic Programming Algorithm for Reservoir Production Optimization 566
25.5 Simulation Results 573
25.6 Concluding Remarks 578
26. A Learning Strategy for Source Tracking in Unstructured Environments 582
Titus Appel, Rafael Fierro, Brandon Rohrer, Ron Lumia, and John Wood
26.1 Introduction 582
26.2 Reinforcement Learning 583
26.3 Light-Following Robot 589
26.4 Simulation Results 592
26.5 Experimental Results 595
26.6 Conclusions and Future Work 599
References 599
INDEX 601
