Title Page
Contents
Chapter 1. Introduction 11
1.1. Image super-resolution 11
1.2. Classification of image super-resolution 13
1.2.1. Single image super-resolution 13
1.2.2. Multi image super-resolution 14
1.2.3. Video super-resolution 15
1.3. Image super-resolution approaches 16
1.3.1. Interpolation based methods (Traditional methods / Image processing methods) 17
1.3.2. Reconstruction methods 17
1.3.3. Learning based methods 19
1.4. Problem statement of single image super-resolution 19
1.5. Contributions 22
Chapter 2. Deep Learning based Super-Resolution 24
2.1. Introduction 25
2.2. Challenges in deep learning based super-resolution 30
2.3. Considerations in model selection: SRResNet 32
2.3.1. General issues in deep learning network 33
2.3.2. Super-resolution residual networks: SRResNet 36
Chapter 3. Image Super-Resolution using Patch Inputs 37
3.1. Super-resolution residual neural network: SRResNet 38
3.2. Motivation of patch input based super-resolution 42
3.3. Patch input-based super-resolution algorithm 50
3.3.1. Patch extraction 50
3.3.2. Image reconstruction from super-resolved patches 52
3.4. Experimental results 53
3.4.1. Experimental setup 53
3.4.2. Simulation results 55
3.5. Problems in patch approach 60
3.6. Conclusion 62
Chapter 4. Padding Methods 64
4.1. Padding algorithms 65
4.2. Partial convolution based padding (PCP) 68
4.3. Performance analysis of Partial Convolution based Padding 70
4.4. Problems in partial convolution based padding 77
4.4.1. Signed PCP (sPCP) 78
4.4.2. Adaptive PCP (aPCP) 80
4.5. Performance analysis of padding algorithms 83
4.6. Conclusion 88
Chapter 5. Training Set Selection 90
5.1. Effects of training set 91
5.2. Training set diversification approaches 92
5.3. Dataset selection using determinantal point process 94
5.3.1. Determinantal point process (definition) 94
5.3.2. k-DPP algorithm 100
5.4. Experimental results 101
5.5. Conclusion 106
Chapter 6. Conclusion and Future Work 110
References 120
Abstract 134
Table 3.1. Computational resources consumption by SRResNet and SRResNetp 59
Table 3.2. Performance comparison input size (asterisk * symbol refers to best result) 59
Table 4.1. Performance Analysis on the border region (padded part) and the inner part 74
Table 4.2. Comparison of the padding schemes. (asterisk * symbol refers to best result, number in parentheses show the number of images where the model is... 75
Table 4.3. Summary of padding methods 83
Table 4.4. Performance comparison: proposed methods vs state-of-the-art methods 85
Table 4.5. Computational resource 87
Table 5.1. Impact of k on model's performance 105
Figure 1.1. Image super-resolution 11
Figure 1.2. Illustration of ill-posed problem. 12
Figure 1.3. Illustration of many to one mapping 15
Figure 1.4. Super-resolution classification 16
Figure 1.5. MSE/PSNR values vs visual perception quality. 21
Figure 2.1. Convolution operation 29
Figure 2.2. A sample VGG loss sketch 32
Figure 2.3. Considerations in deep learning model selection 33
Figure 2.4. A sample skip connection 34
Figure 2.5. Upsampler designs 35
Figure 3.1. Various designs of a skip connection 39
Figure 3.2. Various architectures of Residual Block 40
Figure 3.3. Network architecture of SRResNet 43
Figure 3.4. Training set preparation strategies 46
Figure 3.5. Fixed input size requirement by trained model 48
Figure 3.6. Illustration of patch extraction. 50
Figure 3.7. Illustration of image reconstruction.[원문불량;p.42] 52
Figure 3.8. Impact of local and non-local features. (a) High-resolution image, (b) Super-resolved image by SRResNet, (c) difference of a & b, (d) Super-resolved... 56
Figure 3.9. Impact of local and non-local features. (a) High-resolution image, (b) manually modified image, (c) modified areas, (d) Super-resolved image of a by...[원문불량;p.47] 57
Figure 3.10. Quality of learning. (a) high-resolution image, (b) zoomed in image of a, (c) super-resolved image by SRResNetp, (d) zoomed in image of...[원문불량;p.51] 61
Figure 3.11. Average MSE of Bicubic, SRResNet, and SRResNetp 62
Figure 4.1. Convolution operation with and without zero padding 65
Figure 4.2. Padding schemes 67
Figure 4.3. Propagation of padded pixels toward the center[원문불량;p.59] 69
Figure 4.4. Difference image between reconstructed SR and target HR image[원문불량;p.59] 69
Figure 4.5. Example of PCP 71
Figure 4.6. Assembled images of pixel MSE of super-resolution and high-resolution images. (a) shows the image of MSE values between super-resolution image using... 73
Figure 4.7. Training and validation MSE: SRResNet vs SRResNet-pcp 75
Figure 4.8. Average MSE of SRResNetp and SRResNet-pcp 77
Figure 4.9. Use of fixed ratio matrix for similar sized inputs 78
Figure 4.10. Problem in PCP 78
Figure 4.11. Illustration of signed PCP 80
Figure 4.12. Adaptive PCP 81
Figure 4.13. Impact of local and non-local features. (a) High-resolution image, (b) manually modified image, (c) modified areas, (d) Super-resolved image of... 82
Figure 4.14. Visual comparison between (a) Original high-resolution image, (b) SRGAN, (c) SRResNet-patch input, (d) SRResNet-pcp, (e) Bicubic interpolation, (f)... 88
Figure 4.15. Memory Requirements of various models 89
Figure 5.1. Normal patching scheme 92
Figure 5.2. DPP vs normal set selection 96
Figure 5.3. Diverse set selection using k-DPP 98
Figure 5.4. Comparison of patch selection methods 100
Figure 5.5. Impact of training set quality. 108
Figure 5.6. Impact of data quality on model's convergence 109
Figure 6.1. Padding network 114
Figure 6.2. Super-resolution in frequency domain 114
Figure 6.3. SRResNet super-resolution in frequency domain 116
Figure 6.4. Architecture of a residual network without padding 117
Figure 6.5. Visual results of the residual network without padding scheme 118
Figure 6.6. Augmented MSE 120