Title Page
Abstract
Contents
Chapter 1. Introduction 17
1.1. Motivation 17
1.2. Contribution 23
1.3. Dissertation Organization 25
Chapter 2. Background 26
2.1. NVIDIA Jetson AGX Xavier 26
2.2. NVIDIA TensorRT 27
2.3. Genetic Algorithm 28
2.4. Compositional Performance Analysis 29
2.5. Model-based Design Methodology 30
Chapter 3. Optimization of a Single Deep Learning Application 32
3.1. Overview 32
3.2. Related Work 32
3.2.1. Deep learning Frameworks 33
3.2.2. Optimization For a Single Deep Learning Application 33
3.3. Parallelization Techniques 35
3.3.1. Pre/Post-Processing Pipelining and Parallelization 35
3.3.2. Intra-PE Parallelization 36
3.3.3. Intra-network Pipelining 37
3.3.4. Partial Network Duplication 38
3.3.5. Other Optimization Methods 38
3.4. JEDI Framework 39
3.4.1. Configuration Parameters 41
3.4.2. Application Development 43
3.5. Design Space Exploration 45
3.5.1. Pipeline Cut-point Explorer 47
3.5.2. Parameter Fine-tuner 53
3.6. Experiments 54
3.6.1. Set-Up 54
3.6.2. Design Space Exploration Results 56
3.6.3. Parameter Fine-tuning Results 58
3.6.4. Comparison with Other Methods 59
3.6.5. Experiments with Varying Configurations 64
3.6.6. Analysis and Discussion 67
Chapter 4. Optimization of Multiple Deep Learning Applications under Real-time Constraints 71
4.1. Overview 71
4.2. Related Work 71
4.2.1. Mapping and Scheduling Multiple Applications 72
4.2.2. Running Multiple Deep Learning Applications 74
4.3. System Model 76
4.3.1. Motivational Example 76
4.3.2. Notation 77
4.3.3. Problem Formulation 80
4.4. Proposed Optimization Methodology 81
4.4.1. Step 1: Finding Pareto-optimal Mapping Solutions for Each Application 81
4.4.2. Step 2: Exploring the Mapping Combination 84
4.4.3. Step 3: Tuning Frequencies for Varying Deadline Constraints 92
4.5. Experiments 92
4.5.1. Set-Up 92
4.5.2. Finding Pareto-optimal Mappings of Each Application 93
4.5.3. Exploring Mapping Combination and Tuning Frequencies 94
4.5.4. Real Deployment 102
Chapter 5. Supporting Deep Learning Applications in a Model-based Design Methodology 104
5.1. Overview 104
5.2. Related work 104
5.2.1. Mapping of Multiple Dataflow Applications 105
5.2.2. Mapping of Multiple Deep Learning Applications 106
5.2.3. Integrating Deep Learning Applications into the Model-based Design 106
5.3. System Model 107
5.3.1. Motivational Example 107
5.3.2. Notation and Problem Definition 108
5.4. Proposed Methodology 110
5.4.1. Step 1: Finding the Pareto-optimal Mapping Solutions of Each Deep Learning Application 111
5.4.2. Step 2: Mapping Exploration 114
5.5. Experiments 121
5.5.1. Comparison with a Previous Work 121
5.5.2. Set-up 122
5.5.3. Experimental Results: Motivational Example 122
5.5.4. Experimental Results: Randomly Generated Dataflow Graphs 124
Chapter 6. Conclusion and Future work 127
Bibliography 130
요약 139
Table 3.1. Main virtual methods for user-implemented deep learning applications in JEDI 44
Table 3.2. Options for mapping on the target platform 45
Table 3.3. The labels and the number of layers of benchmark applications. 55
Table 3.4. The search time and range of the network pipelining heuristic 57
Table 3.5. Fine-tuned configurations of the selected cut-points from our methodology 58
Table 3.6. FPS comparison of using the solution obtained by the profile-based method as the initial solution 63
Table 3.7. Comparison of exploration methods 64
Table 3.8. Comparison of the best result of reused cut-points from 416x416 and the re-explored cut-points with our fast heuristic search 65
Table 3.9. Comparison of the best result of reused cut-points from batch 1 and the re-explored cut-points with our fast heuristic search 66
Table 3.10. Inference time comparison between baseline method and the found mapping by the proposed method 67
Table 3.11. Inference time comparison between baseline and the found mapping by the proposed method 68
Table 4.1. Comparison with the related works of running multiple deep learning applications 74
Table 4.2. Notations for system model and problem definition 78
Table 4.3. Options for intra-network pipelining 82
Table 4.4. The benchmark networks and the volume of design space for step 1 to find the Pareto-optimal mappings. 94
Table 4.5. Information on three different system behaviors: which applications are per-formed in each state 94
Table 4.6. Cases for a combination of applications in FSMs 96
Table 4.7. The frequency range used in the exploration 96
Table 4.8. Comparison of the maximum achievable FPS among methods 97
Table 4.9. Mapping and frequency tuning result for FSM-A from the proposed method 97
Table 4.10. Throughput constraint (FPS) variation 99
Table 4.11. Average fitness evaluation time and end-to-end execution time during 50 generations in GA 101
Table 5.1. Notations used in a system model 111
Table 5.2. Mapping options for pipelining of a DL application 112
Table 5.3. Benchmark networks and the number of mapping candidates obtained by step 1 121
Table 5.4. Mappable processors of tasks in dataflow applications 121
Figure 1.1. Three topics for system-level optimization on AI hardware platforms 18
Figure 2.1. The example composition of kernels 27
Figure 2.2. The workflow of generating execution context in TensorRT 27
Figure 2.3. An example schedule of deep learning inference 28
Figure 2.4. Schematic diagram of compositional performance analysis 29
Figure 2.5. Overall flow of the model-based embedded software design 31
Figure 3.1. Four parallelization techniques 35
Figure 3.2. Schedule diagrams of inference steps with pre/post-processing pipelining 35
Figure 3.3. Schedule diagrams of inference body parallelization 37
Figure 3.4. Workflow of the proposed inference framework 40
Figure 3.5. An example segment of JEDI configuration file 41
Figure 3.6. The proposed optimization process with three design space exploration modules 45
Figure 3.7. Illustration of the proposed heuristic with an example 49
Figure 3.8. FPS comparison among options on FP16 and INT8 precision 56
Figure 3.9. FPS, energy comparison, and CPU/GPU utilization among four methods with FP16 precision 60
Figure 3.10. FPS, energy comparison, and CPU/GPU utilization among four methods with INT8 precision 61
Figure 3.11. Comparison with interleaved execution on different processors 62
Figure 3.12. Gantt charts for different mappings of the Yolov4 network 68
Figure 3.13. Gantt chart based on estimated layer-wise execution time for a found map- ping by the proposed method 69
Figure 4.1. Motivational example: patrol robot 76
Figure 4.2. Pipelining of the DL application 78
Figure 4.3. Overall flow of the proposed mapping methodology 81
Figure 4.4. The overview of step 1 82
Figure 4.5. The overview of step 2 84
Figure 4.6. Fitness ratio as the FPS constraint varies for FSM-A 100
Figure 4.7. Average fitness ratio over the FPS constraint variation for all benchmarks 100
Figure 4.8. The run time management following scenarios 101
Figure 4.9. Average energy ratio over the FPS constraint variation, measured after real deployment 102
Figure 5.1. A motivational example 108
Figure 5.2. Task/Sub-task definition on the deep learning application specified by SDK 109
Figure 5.3. Overall flow of the model-based embedded software design and the proposed extension 112
Figure 5.4. Chromosome structures: (a) for step1, (b) for Heuristic+GA method in step2, and (c) for Entire-GA method in step2. 114
Figure 5.5. Procedure of the proposed mapping exploration technique 115
Figure 5.6. Comparison of three methods for the motivational example: Heuristic+GA, Entire-GA, and Baseline 122
Figure 5.7. Comparison of three methods with four randomly generated dataflow applications. 124
Figure 5.8. Comparison of three methods with eight randomly generated dataflow applications. 124
Figure 5.9. Comparison of three methods with sixteen randomly generated dataflow applications. 125
Algorithm 1. Pseudo code for genetic algorithm 28
Algorithm 2. Pseudo code for global search heuristic 48
Algorithm 3. Pseudo for local search heuristic 51
Algorithm 4. Pseudo code for chromosome evaluation 85
Algorithm 5. Pseudo code for mapping stages to PE 86
Algorithm 6. Pseudo code for frequencies decision 87
Algorithm 7. Pseudo code of mapping heuristic for dataflow tasks 118