Title Page
Contents
ABSTRACT 9
요약 10
Chapter 1. Introduction 11
Chapter 2. Related Works 15
2.1. Data-Free Knowledge Distillation 15
2.2. Learning from Noisy Labels 17
2.3. Generative Models 18
Chapter 3. Methodology 20
3.1. Framework of Generator-Based DFKD 20
3.1.1. Our Findings. 21
3.2. Revisiting Class-Prior in DFKD 21
3.2.1. Lower Sample Diversity. 22
3.2.2. Incomplete Quality Control. 23
3.3. Proposed Method 25
3.3.1. Generator Loss Without Class-Prior. 25
3.3.2. Quality Control via Sample Selection. 26
3.3.3. Overall Process of TA-DFKD. 28
Chapter 4. Experiments 30
4.1. Environment 30
4.1.1. Datasets and Compared Methods. 30
4.1.2. Training Details. 31
4.1.3. Evaluation Metrics. 31
4.2. Experimental Results 32
4.2.1. Performance Comparison. 32
4.2.2. Teacher-Agnostic Behavior. 32
4.2.3. Ablation Study. 35
4.2.4. Results of Tiny-ImageNet. 36
4.3. Implementation Details 37
4.3.1. How to Train Various Teacher Models 37
4.3.2. Details of Implementation and Hyperparameters of TA-DFKD 38
Chapter 5. Conclusion 39
Bibliography 40
이력서 46
TABLE 2.1. Summary of the existing DFKD methods, DAFL [5], DFAD [8], ADI [33], CMI [9], PRE-DFKD [2] and TA-DFKD (ours), in terms of using three major... 16
TABLE 3.1. FID scores using five different ResNet-34 teacher models on CIFAR-10. 23
TABLE 4.1. DFKD performance comparison using 5 teacher models trained on CIFAR-10 (top) and CIFAR-100 (bottom). 33
TABLE 4.2. Differences in performance between teachers and student models (Teacher Acc. - Student Peak Acc. (accmax)).[이미지참조] 34
TABLE 4.3. Differences between the peak accuracy and converging accuracy within each student model (Peak Acc. (accmax) - Converging Acc. (acclast[k]).[이미지참조] 35
TABLE 4.4. Ablation study on CIFAR-10 and CIFAR-100, where (1) baseline is the standard DFKD framework involving all the three loss terms, (2) w/o class-prior is the... 36
TABLE 4.5. Results of Tiny-ImageNet in two teacher models 36
FIGURE 1.1. (a) Training curves of student models during distillation of two different well-trained teacher models on MNIST when using one or both of class-prior (DAFL)... 12
FIGURE 3.1. FID scores using a pretrained ResNet-34 model on CIFAR-10 with class-prior's intensity values from 0 to 1. 22
FIGURE 3.2. Images of Airplane (top) and Dog (bottom) generated by each trained version of the generator with or without class-prior for a pretrained ResNet-34 model... 22
FIGURE 3.3. (a) 2D visualization of feature vectors corresponding to real data and synthetic data generated by a generator trained using class-prior without the adversarial... 24
FIGURE 3.4. Visualization of before and after sample selection from real validation data and synthetic samples in the teacher model ResNet-34 on CIFAR-10. 27
FIGURE 3.5. Overview of the proposed TA-DFKD method. 29
FIGURE 4.1. Box-plots of performance differences between teacher and student in the CIFAR datasets. 34
FIGURE 4.2. Differences between accmax and acclast[k] in the CIFAR datasets.[이미지참조] 35