Foreword xvi
by Vipin Kumar, Regents Professor, University of Minnesota
Acknowledgments xvii
List of Contributors xviii
List of Acronyms xxiv
1 Introduction 1
Gustau Camps-Valls, Xiao Xiang Zhu, Devis Tuia, and Markus Reichstein
1.1 A Taxonomy of Deep Learning Approaches 2
1.2 Deep Learning in Remote Sensing 3
1.3 Deep Learning in Geosciences and Climate 7
1.4 Book Structure and Roadmap 9
Part I Deep Learning to Extract Information from Remote Sensing Images 13
2 Learning Unsupervised Feature Representations of Remote Sensing Data with Sparse Convolutional Networks 15
Jose E. Adsuara, Manuel Campos-Taberner, Javier García-Haro, Carlo Gatta, Adriana Romero, and Gustau Camps-Valls
2.1 Introduction 15
2.2 Sparse Unsupervised Convolutional Networks 17
2.2.1 Sparsity as the Guiding Criterion 17
2.2.2 The EPLS Algorithm 18
2.2.3 Remarks 18
2.3 Applications 19
2.3.1 Hyperspectral Image Classification 19
2.3.2 Multisensor Image Fusion 21
2.4 Conclusions 22
3 Generative Adversarial Networks in the Geosciences 24
Gonzalo Mateo-García, Valero Laparra, Christian Requena-Mesa, and Luis Gómez-Chova
3.1 Introduction 24
3.2 Generative Adversarial Networks 25
3.2.1 Unsupervised GANs 25
3.2.2 Conditional GANs 26
3.2.3 Cycle-consistent GANs 27
3.3 GANs in Remote Sensing and Geosciences 28
3.3.1 GANs in Earth Observation 28
3.3.2 Conditional GANs in Earth Observation 30
3.3.3 CycleGANs in Earth Observation 30
3.4 Applications of GANs in Earth Observation 31
3.4.1 Domain Adaptation Across Satellites 31
3.4.2 Learning to Emulate Earth Systems from Observations 33
3.5 Conclusions and Perspectives 36
4 Deep Self-taught Learning in Remote Sensing 37
Ribana Roscher
4.1 Introduction 37
4.2 Sparse Representation 38
4.2.1 Dictionary Learning 39
4.2.2 Self-taught Learning 40
4.3 Deep Self-taught Learning 40
4.3.1 Application Example 43
4.3.2 Relation to Deep Neural Networks 44
4.4 Conclusion 45
5 Deep Learning-based Semantic Segmentation in Remote Sensing 46
Devis Tuia, Diego Marcos, Konrad Schindler, and Bertrand Le Saux
5.1 Introduction 46
5.2 Literature Review 47
5.3 Basics on Deep Semantic Segmentation: Computer Vision Models 49
5.3.1 Architectures for Image Data 49
5.3.2 Architectures for Point-clouds 52
5.4 Selected Examples 55
5.4.1 Encoding Invariances to Train Smaller Models: The example of Rotation 55
5.4.2 Processing 3D Point Clouds as a Bundle of Images: SnapNet 59
5.4.3 Lake Ice Detection from Earth and from Space 62
5.5 Concluding Remarks 66
6 Object Detection in Remote Sensing 67
Jian Ding, Jinwang Wang, Wen Yang, and Gui-Song Xia
6.1 Introduction 67
6.1.1 Problem Description 67
6.1.2 Problem Settings of Object Detection 69
6.1.3 Object Representation in Remote Sensing 69
6.1.4 Evaluation Metrics 69
6.1.4.1 Precision-Recall Curve 70
6.1.4.2 Average Precision and Mean Average Precision 71
6.1.5 Applications 71
6.2 Preliminaries on Object Detection with Deep Models 72
6.2.1 Two-stage Algorithms 72
6.2.1.1 R-CNNs 72
6.2.1.2 R-fcn 73
6.2.2 One-stage Algorithms 73
6.2.2.1 Yolo 73
6.2.2.2 Ssd 73
6.3 Object Detection in Optical RS Images 75
6.3.1 Related Works 75
6.3.1.1 Scale Variance 75
6.3.1.2 Orientation Variance 75
6.3.1.3 Oriented Object Detection 75
6.3.1.4 Detecting in Large-size Images 76
6.3.2 Datasets and Benchmark 77
6.3.2.1 Dota 77
6.3.2.2 VisDrone 77
6.3.2.3 Dior 77
6.3.2.4 xView 77
6.3.3 Two Representative Object Detectors in Optical RS Images 78
6.3.3.1 Mask OBB 78
6.3.3.2 RoI Transformer 82
6.4 Object Detection in SAR Images 86
6.4.1 Challenges of Detection in SAR Images 86
6.4.2 Related Works 86
6.4.3 Datasets and Benchmarks 88
6.5 Conclusion 89
7 Deep Domain Adaptation in Earth Observation 90
Benjamin Kellenberger, Onur Tasar, Bharath Bhushan Damodaran, Nicolas Courty, and Devis Tuia
7.1 Introduction 90
7.2 Families of Methodologies 91
7.3 Selected Examples 93
7.3.1 Adapting the Inner Representation 93
7.3.2 Adapting the Inputs Distribution 97
7.3.3 Using (few, well chosen) Labels from the Target Domain 100
7.4 Concluding remarks 104
8 Recurrent Neural Networks and the Temporal Component 105
Marco Körner and Marc Rußwurm
8.1 Recurrent Neural Networks 106
8.1.1 Training RNNs 107
8.1.1.1 Exploding and Vanishing Gradients 107
8.1.1.2 Circumventing Exploding and Vanishing Gradients 109
8.2 Gated Variants of RNNs 111
8.2.1 Long Short-term Memory Networks 111
8.2.1.1 The Cell State c t and the Hidden State h t 112
8.2.1.2 The Forget Gate f t 112
8.2.1.3 The Modulation Gate V T and the Input Gate I T 112
8.2.1.4 The Output Gate o t 112
8.2.1.5 Training LSTM Networks 113
8.2.2 Other Gated Variants 113
8.3 Representative Capabilities of Recurrent Networks 114
8.3.1 Recurrent Neural Network Topologies 114
8.3.2 Experiments 115
8.4 Application in Earth Sciences 117
8.5 Conclusion 118
9 Deep Learning for Image Matching and Co-registration 120
Maria Vakalopoulou, Stergios Christodoulidis, Mihir Sahasrabudhe, and Nikos Paragios
9.1 Introduction 120
9.2 Literature Review 123
9.2.1 Classical Approaches 123
9.2.2 Deep Learning Techniques for Image Matching 124
9.2.3 Deep Learning Techniques for Image Registration 125
9.3 Image Registration with Deep Learning 126
9.3.1 2D Linear and Deformable Transformer 126
9.3.2 Network Architectures 127
9.3.3 Optimization Strategy 128
9.3.4 Dataset and Implementation Details 129
9.3.5 Experimental Results 129
9.4 Conclusion and Future Research 134
9.4.1 Challenges and Opportunities 134
9.4.1.1 Dataset with Annotations 134
9.4.1.2 Dimensionality of Data 135
9.4.1.3 Multitemporal Datasets 135
9.4.1.4 Robustness to Changed Areas 135
10 Multisource Remote Sensing Image Fusion 136
Wei He, Danfeng Hong, Giuseppe Scarpa, Tatsumi Uezato, and Naoto Yokoya
10.1 Introduction 136
10.2 Pansharpening 137
10.2.1 Survey of Pansharpening Methods Employing Deep Learning 137
10.2.2 Experimental Results 140
10.2.2.1 Experimental Design 140
10.2.2.2 Visual and Quantitative Comparison in Pansharpening 140
10.3 Multiband Image Fusion 143
10.3.1 Supervised Deep Learning-based Approaches 143
10.3.2 Unsupervised Deep Learning-based Approaches 145
10.3.3 Experimental Results 146
10.3.3.1 Comparison Methods and Evaluation Measures 146
10.3.3.2 Dataset and Experimental Setting 146
10.3.3.3 Quantitative Comparison and Visual Results 147
10.4 Conclusion and Outlook 148
11 Deep Learning for Image Search and Retrieval in Large Remote Sensing Archives 150
Gencer Sumbul, Jian Kang, and Begüm Demir
11.1 Introduction 150
11.2 Deep Learning for RS CBIR 152
11.3 Scalable RS CBIR Based on Deep Hashing 156
11.4 Discussion and Conclusion 159
Acknowledgement 160
Part II Making a Difference in the Geosciences with Deep Learning 161
12 Deep Learning for Detecting Extreme Weather Patterns 163
Mayur Mudigonda, Prabhat Ram, Karthik Kashinath, Evan Racah, Ankur Mahesh, Yunjie Liu, Christopher Beckham, Jim Biard, Thorsten Kurth, Sookyung Kim, Samira Kahou, Tegan Maharaj, Burlen Loring, Christopher Pal, Travis O'Brien, Kenneth E. Kunkel, Michael F. Wehner, and William D. Collins
12.1 Scientific Motivation 163
12.2 Tropical Cyclone and Atmospheric River Classification 166
12.2.1 Methods 166
12.2.2 Network Architecture 167
12.2.3 Results 169
12.3 Detection of Fronts 170
12.3.1 Analytical Approach 170
12.3.2 Dataset 171
12.3.3 Results 172
12.3.4 Limitations 174
12.4 Semi-supervised Classification and Localization of Extreme Events 175
12.4.1 Applications of Semi-supervised Learning in Climate Modeling 175
12.4.1.1 Supervised Architecture 176
12.4.1.2 Semi-supervised Architecture 176
12.4.2 Results 176
12.4.2.1 Frame-wise Reconstruction 176
12.4.2.2 Results and Discussion 178
12.5 Detecting Atmospheric Rivers and Tropical Cyclones Through Segmentation Methods 179
12.5.1 Modeling Approach 179
12.5.1.1 Segmentation Architecture 180
12.5.1.2 Climate Dataset and Labels 181
12.5.2 Architecture Innovations: Weighted Loss and Modified Network 181
12.5.3 Results 183
12.6 Challenges and Implications for the Future 184
12.7 Conclusions 185
13 Spatio-temporal Autoencoders in Weather and Climate Research 186
Xavier-Andoni Tibau, Christian Reimers, Christian Requena-Mesa, and Jakob Runge
13.1 Introduction 186
13.2 Autoencoders 187
13.2.1 A Brief History of Autoencoders 188
13.2.2 Archetypes of Autoencoders 189
13.2.3 Variational Autoencoders (VAE) 191
13.2.4 Comparison Between Autoencoders and Classical Methods 192
13.3 Applications 193
13.3.1 Use of the Latent Space 193
13.3.1.1 Reduction of Dimensionality for the Understanding of the System Dynamics and its Interactions 195
13.3.1.2 Dimensionality Reduction for Feature Extraction and Prediction 199
13.3.2 Use of the Decoder 199
13.3.2.1 As a Random Sample Generator 201
13.3.2.2 Anomaly Detection 201
13.3.2.3 Use of a Denoising Autoencoder (DAE) Decoder 202
13.4 Conclusions and Outlook 203
14 Deep Learning to Improve Weather Predictions 204
Peter D. Dueben, Peter Bauer, and Samantha Adams
14.1 Numerical Weather Prediction 204
14.2 How Will Machine Learning Enhance Weather Predictions? 207
14.3 Machine Learning Across the Workflow of Weather Prediction 208
14.4 Challenges for the Application of ML in Weather Forecasts 213
14.5 The Way Forward 216
15 Deep Learning and the Weather Forecasting Problem: Precipitation Nowcasting 218
Zhihan Gao, Xingjian Shi, Hao Wang, Dit-Yan Yeung, Wang-chun Woo, and Wai-Kin Wong
15.1 Introduction 218
15.2 Formulation 220
15.3 Learning Strategies 221
15.4 Models 223
15.4.1 FNN-based Odels 223
15.4.2 RNN-based Models 225
15.4.3 Encoder-forecaster Structure 226
15.4.4 Convolutional LSTM 226
15.4.5 ConvLSTM with Star-shaped Bridge 227
15.4.6 Predictive RNN 228
15.4.7 Memory in Memory Network 229
15.4.8 Trajectory GRU 231
15.5 Benchmark 233
15.5.1 HKO-7 Dataset 234
15.5.2 Evaluation Methodology 234
15.5.3 Evaluated Algorithms 235
15.5.4 Evaluation Results 236
15.6 Discussion 236
Appendix 238
Acknowledgement 239
16 Deep Learning for High-dimensional Parameter Retrieval 240
David Malmgren-Hansen
16.1 Introduction 240
16.2 Deep Learning Parameter Retrieval Literature 242
16.2.1 Land 242
16.2.2 Ocean 243
16.2.3 Cryosphere 244
16.2.4 Global Weather Models 244
16.3 The Challenge of High-dimensional Problems 244
16.3.1 Computational Load of CNNs 247
16.3.2 Mean Square Error or Cross-entropy Optimization? 249
16.4 Applications and Examples 250
16.4.1 Utilizing High-Dimensional Spatio-spectral Information with CNNs 250
16.4.2 The Effect of Loss Functions in Retrieval of Sea Ice Concentrations 253
16.5 Conclusion 257
17 A Review of Deep Learning for Cryospheric Studies 258
Lin Liu
17.1 Introduction 258
17.2 Deep-learning-based Remote Sensing Studies of the Cryosphere 260
17.2.1 Glaciers 260
17.2.2 Ice Sheet 261
17.2.3 Snow 262
17.2.4 Permafrost 263
17.2.5 Sea Ice 264
17.2.6 River Ice 265
17.3 Deep-learning-based Modeling of the Cryosphere 265
17.4 Summary and Prospect 266
Appendix: List of Data and Codes 267
18 Emulating Ecological Memory with Recurrent Neural Networks 269
Basil Kraft, Simon Besnard, and Sujan Koirala
18.1 Ecological Memory Effects: Concepts and Relevance 269
18.2 Data-driven Approaches for Ecological memory Effects 270
18.2.1 A Brief Overview of Memory Effects 270
18.2.2 Data-driven Methods for Memory Effects 271
18.3 Case Study: Emulating a Physical Model Using Recurrent Neural Networks 272
18.3.1 Physical Model Simulation Data 272
18.3.2 Experimental Design 273
18.3.3 RNN Setup and Training 274
18.4 Results and Discussion 276
18.4.1 The Predictive Capability Across Scales 276
18.4.2 Prediction of Seasonal Dynamics 279
18.5 Conclusions 281
Part III Linking Physics and Deep Learning Models 283
19 Applications of Deep Learning in Hydrology 285
Chaopeng Shen and Kathryn Lawson
19.1 Introduction 285
19.2 Deep Learning Applications in Hydrology 286
19.2.1 Dynamical System Modeling 286
19.2.1.1 Large-scale Hydrologic Modeling with Big Data 286
19.2.1.2 Data-limited LSTM Applications 290
19.2.2 Physics-constrained Hydrologic Machine Learning 292
19.2.3 Information Retrieval for Hydrology 293
19.2.4 Physically-informed Machine Learning for Subsurface Flow and Reactive Transport Modeling 294
19.2.5 Additional Observations 296
19.3 Current Limitations and Outlook 296
20 Deep Learning of Unresolved Turbulent Ocean Processes in Climate Models 298
Laure Zanna and Thomas Bolton
20.1 Introduction 298
20.2 The Parameterization Problem 299
20.3 Deep Learning Parameterizations of Subgrid Ocean Processes 300
20.3.1 Why DL for Subgrid Parameterizations? 300
20.3.2 Recent Advances in DL for Subgrid Parameterizations 300
20.4 Physics-aware Deep Learning 301
20.5 Further Challenges ahead for Deep Learning Parameterizations 303
21 Deep Learning for the Parametrization of Subgrid Processes in Climate Models 307
Pierre Gentine, Veronika Eyring, and Tom Beucler
21.1 Introduction 307
21.2 Deep Neural Networks for Moist Convection (Deep Clouds) Parametrization 309
21.3 Physical Constraints and Generalization 312
21.4 Future Challenges 314
22 Using Deep Learning to Correct Theoretically-derived Models 315
PeterA.G.Watson
22.1 Experiments with the Lorenz '96 System 317
22.1.1 The Lorenz'96 Equations and Coarse-scale Models 318
22.1.1.1 Theoretically-derived Coarse-scale Model 318
22.1.1.2 Models with ANNs 319
22.1.2 Results 320
22.1.2.1 Single-timestep Tendency Prediction Errors 320
22.1.2.2 Forecast and Climate Prediction Skill 321
22.1.3 Testing Seamless Prediction 324
22.2 Discussion and Outlook 324
22.2.1 Towards Earth System Modeling 325
22.2.2 Application to Climate Change Studies 326
22.3 Conclusion 327
23 Outlook 328
Markus Reichstein, Gustau Camps-Valls, Devis Tuia, and Xiao Xiang Zhu
Bibliography 331
Index 401
