Title Page
Abstract
Contents
Chapter 1. Introduction 20
Chapter 2. Generative Modeling and Epistemic Uncertainty Quantification 24
2.1. Uncertainty Quantification in Machine Learning 25
2.2. Probability is Relative Epistemic Uncertainty 27
Chapter 3. Normalized Autoencoders: A Probabilistic View on Autoencoder-Based Anomaly Detection 30
3.1. Introduction 30
3.2. Background 33
3.2.1. Autoencoders 33
3.2.2. Energy-based Models 35
3.2.3. Outlier Reconstruction 36
3.3. Normalized Autoencoders 39
3.3.1. Definition 39
3.3.2. Remarks 40
3.4. On-Manifold Initialization 41
3.4.1. Failure Modes of CD and PCD 43
3.4.2. On-Manifold Initialization 45
3.5. Related Work 46
3.6. Experiments 48
3.6.1. Technicalities for NAE Training 48
3.6.2. 2D Density Estimation 49
3.6.3. Outlier Detection 50
3.6.4. Sample Generation 52
3.7. Discussion 54
3.7.1. Comparison to Other EBMs 54
3.7.2. Likelihood-based Outlier Detection 55
3.7.3. Analytic Solution for Linear Case 56
3.8. Conclusion 58
Chapter 4. Manifold Projection-Diffusion Recovery: Leveraging Manifold Structure in Energy-Based Model Training 59
4.1. Introduction 59
4.2. Preliminaries 63
4.3. Manifold Projection-Diffusion Recovery 64
4.3.1. Manifold Projection-Diffusion 65
4.3.2. Manifold Projection-Diffusion Recovery Likelihood 66
4.3.3. Consistency of MPDR 67
4.3.4. Two-Stage Sampling 69
4.3.5. Perturbation Ensemble 69
4.3.6. Energy Function Design 70
4.4. Experiment 72
4.4.1. Implementation of MPDR 72
4.4.2. 2D Density Estimation 73
4.4.3. Image Out-of-Distribution Detection 74
4.4.4. Acoustic Anomaly Detection 77
4.4.5. Ablation Study 78
4.4.6. Mode Collapse 82
4.4.7. Comparison to Score-Based OOD Detection 83
4.5. Conclusion 83
4.6. Experimental Details and Additional Results 86
4.6.1. MNIST 86
4.6.2. Sensitivity to σ 88
4.6.3. CIFAR-10 OOD Detection 89
4.6.4. CIFAR-100 OOD Detection on Pretrained Representation 92
4.6.5. Acoustic Anomaly Detection 93
4.7. Empirical Guidelines for Implementing MPDR 96
Chapter 5. Generating Adversarial Outliers with Energy-Based Models 97
5.1. Introduction 97
5.2. Related Work 100
5.3. Robustness in OOD Detection 102
5.3.1. Out-of-Distribution Detection 102
5.3.2. Robustness of OOD Detectors 103
5.4. Adversarial Generation of Outliers on Manifolds 104
5.4.1. Outlier Manifolds 105
5.4.2. Adversarial Generation via MCMC Ensemble 107
5.5. Experiments 109
5.5.1. Experimental Settings 109
5.5.2. CIFAR-10 Experiment 116
5.5.3. RImgNet Experiment 117
5.6. Discussion 118
5.7. Conclusion 120
Chapter 6. Generative Gaussian Process: Gaussian Process as an Energy-Based Model 121
6.1. Introduction 121
6.2. Gaussian Processes Are Density Estimators 123
6.3. Density Estimators in Gaussian Processes May Mislead 126
6.4. Generative Gaussian Process Regression 127
6.5. Bayesian Regressors Are Approximately Density Estimators 128
6.6. Experiment 130
6.6.1. Oversmoothed Predictive Variance 130
6.7. Active Learning 130
Chapter 7. Conclusion 133
7.1. Summary and Key Takeaways 133
7.2. Future Directions 134
Bibliography 136
국문초록 149
Table 3.1. MNIST hold-out class detection AUC scores. The intervals denote the standard error of mean after 10 training runs. 47
Table 3.2. OOD detection performance in AUC. 54
Table 3.3. FID core of 50,000 images generated from a model trained on CelebA 64x64. A low FID score indicates that the generated images have similar statis-... 55
Table 4.1. MNIST hold-out digit detection. Performance is measured in AUPR. Standard deviation of AUPR is computed over the last 10 epochs. The largest... 71
Table 4.2. MNIST OOD detection performance measured in AUPR. We test models from hold-out digit 9 experiment (Table 1). The overall performance is... 72
Table 4.3. OOD detection with CIFAR-10 as in-distribution. AUROC values are presented. The largest value in the column is marked as boldface, and the... 75
Table 4.4. OOD detection on pretrained ViT-B_16 representation with CIFAR-100 as in-distribution. Performance is measured in AUROC. 76
Table 4.5. Sensitivity to σ. MPDR-S is run with an autoencoder with varying values of noise magnitude σ. AUPR against various outlier datasets are pre-... 80
Table 4.6. Sensitivity to Dz. MPDR-S is run with an autoencoder with varying values of Dz. AUPR against various outlier datasets are presented. For MNIST...[이미지참조] 80
Table 4.7. Acoustic anomaly detection on DCASE2020 Track 2 Dataset. AUROC and pAUROC (in parenthesis) are displayed per cent. 81
Table 4.8. CIFAR-10 OOD detection experiment with the score norm as OOD score. 84
Table 4.9. Hyperparameters for LMC. Latent chain hyperparameters are denoted by Z and X indicates visible chain hyperparameters. "scale (γ)" refers to the multiplicative scale factor on the perturbation probability. 85
Table 4.10. Convolutional neural network architectures used in experiments. The parenthesis following the network name indicates the activation function used... 88
Table 4.11. Sensitivity of γ, demonstrated in CIFAR-100 experiment. AUROC values are displayed. 93
Table 5.1. CIFAR-10 experiment. Clean indicates the test split of a test OOD dataset. AUC scores are evaluated using 10,000 inliers and 1,000 outliers. Min-... 108
Table 5.2. RImgNet Experiment. AGOM is applied to 4 OOD detectors. Other conditions are the same as Table 5.1. 113
Table 5.3. Robustness to l∞ attack, measured in AUC.[이미지참조] 118
Figure 2.1. Illustrations of aleatoric uncertainty and epistemic uncertainty. 25
Figure 2.2. An illustration of the discrete input space regression example. 27
Figure 3.1. Examples of reconstructed outliers. The last two rows show the reconstructions from a conventional autoencoder (AE) and NAE. Both autoencoders... 31
Figure 3.2. AE and NAE trained on a bi-modal distribution. Here, NAE is trained with its decoder fixed. The green lines denotes the decoder manifolds.... 37
Figure 3.3. Detecting hold-out digit 9 from the rest of MNIST. Reconstruction errors and AUC scores are shown across multiple values of Dz. The error bars...[이미지참조] 39
Figure 3.4. Density estimates and negative samples from NAEs trained by various approximate sampling methods. The generated samples (blue dots) are... 42
Figure 3.5. An illustration of the on-manifold initialization. The one-dimensional latent space Z and the two-dimensional input space X are shown. The red star is... 44
Figure 3.6. Estimating 8 Gaussians using various autoencoders. The density of an autoencoder (AE) is computed from Eq. (3.10). AE gives a significant amount... 49
Figure 3.7. MNIST hold-out class detection examples from four different hold-out settings (1, 4, 7 and 9). The bottom two rows depict the reconstructions from... 52
Figure 3.8. More samples from NAE trained on CelebA 64×64. While most of the samples are visually sensible, a few generation failures can be spotted.... 53
Figure 3.9. Sampling with NAEs trained on MNIST and CelebA 64×64. (z₀) The random initialization of the latent chain. We visualize fd(z₀). (OMI) Images...[이미지참조] 56
Figure 4.1. (Left) An illustration of Manifold Projection-Diffusion. A datum x is first projected into the latent space Z through the encoder fₑ(x) and... 61
Figure 4.2. Negative sample generation process in MDPR. 66
Figure 4.3. 2D density estimation with a scalar energy function (left) and an autoencoder-based energy function (right). 68
Figure 4.4. Mode collapse experiment. (First) Samples drawn from 25 Gaussians. (Second) Samples from GAN. The figure is adopted from[2]. (Third) Samples... 82
Figure 4.5. Visualization of ||∇ₓlog p(x)|| from a vanilla EBM (left) and MPDR (right). 83
Figure 5.1. Illustration of outlier manifolds. Real images are highlighted with frames, and synthetic images are shown without frame. An instance-conditional... 98
Figure 5.2. Two classes of outlier manifolds considered in AGOM. manifold and is chosen from practical considerations over other options. An...
Figure 5.3. The images that Glow believes to be CIFAR-10, synthesize by AGOM. Glow trained on CIFAR-10 has an blind spot of misclassifying low-... 111
Figure 5.4. Adversarial samples from GAN outlier manifold. A subset of OOD detectors are shown due to the space contraint. 115
Figure 5.5. Adversarial samples from AGOM in RImgNet experiment. More examples can be found in Appendix. (Top two rows) Affine (Bottom) GAN.... 117
Figure 6.1. The connection between the predictive variance of GPR and density estimation. The predictive variance of GPR is decomposed into epistemic and... 123
Figure 6.2. 1D regression example where epistemic uncertainty quantification of vanilla GPR fails. 125
Figure 6.3. Generative GP in 2D. (First column) The true data distribution and the true function to be predicted. Data points are visualized as dots. (Second... 131
Figure 6.4. Active learning experiment. (Upper left) The distribution of unla- belled data. (Upper right) Active learning result averaged over 100 runs. (Lower... 132