Title Page
Abstract
Contents
List of Abbreviations 16
Chapter 1. Introduction 18
1.1. Patent Classification 19
1.2. Challenges of Patent Analysis 19
1.3. Research Contributions 21
1.4. Structure of the Dissertation 23
Chapter 2. Background Knowledge and Related Work 26
2.1. Text Representation and Classification 27
2.2. HMTC Approaches 29
2.2.1. Flat Classification Approaches 30
2.2.2. Local Classification Approaches 32
2.2.3. Global Classification Approaches 33
2.3. Conclusion 35
Chapter 3. Comparison and Analysis of Embedding Methods for Patent Documents 36
3.1. Various Word Embedding Approaches 37
3.1.1. Word2vec 37
3.1.2. FastText 37
3.1.3. GloVe 38
3.1.4. GPT-2 38
3.2. Feature Extraction 39
3.3. Experiments 40
3.3.1. Data 40
3.3.2. Evaluation Metrics 41
3.3.3. Experimental Setup 42
3.3.4. Experimental Results 44
3.4. Conclusion 44
Chapter 4. PatentNet 46
4.1. Pre-trained Language Models 47
4.1.1. Autoregressive Language Modeling 47
4.1.2. Transformer 48
4.1.3. Transformer-XL 49
4.1.4. BERT 49
4.1.5. XLNet 50
4.1.6. RoBERTa 51
4.1.7. ELECTRA 52
4.2. Multi-label Patent Classification Problem 53
4.2.1. Fine-tuning Process 53
4.2.2. Data Pre-processing 54
4.2.3. Patent Representation 55
4.2.4. Classification Layer 56
4.3. Experiment 57
4.3.1. Dataset 57
4.3.2. Evaluation Measures 59
4.3.3. Baselines and Experiment Setup 62
4.3.4. Experimental Results and Discussion 67
4.4. Conclusions 81
Chapter 5. Hierarchical Multi-label Patent Classification 84
5.1. HMTC Problem for Patents 84
5.2. HiPatnetNet Model 85
5.2.1. Text Encoder 87
5.2.2. Structure Encoder 87
5.2.3. Hierarchical Prior Knowledge 88
5.2.4. Hierarchy-GCN 88
5.2.5. Text Feature Propagation Approach 89
5.3. Experiment 90
5.3.1. Dataset and Evaluation Measures 90
5.3.2. Baselines and Experimental Setup 91
5.3.3. Experimental Results and Discussion 92
5.3.4. Conclusion 94
Chapter 6. Summary, Conclusions, and Future Work 96
Bibliography 100
Table 3.1. Experimental results 43
Table 4.1. Description of USPTO-2M 58
Table 4.2. Description of M-patent 59
Table 4.3. Pre-trained models name and details 66
Table 4.4. Hyperparameter setting for fine-tuning various pre-trained models. 67
Table 4.5. Experiments on M-patent dataset 71
Table 4.6. Experiments on the USPTO-2M dataset 72
Table 5.1. Description of datasets. |m| is the no. of labels, Avg.(|mi|) is the average no. of labels...[이미지참조] 91
Table 5.2. Experiments HiPatentNet 93
Figure 2.1. Example of three local classifier approaches 33
Figure 3.1. The CBOW and Skip-gram model architecture 38
Figure 3.2. The CNN architecture for text classification 40
Figure 4.1. An overview of RTD in ELECTRA 52
Figure 4.2. Preview of finue-tuning process using pre-trained language models 54
Figure 4.3. Number of words statistics of the datasets 62
Figure 4.4. Performance of the pre-trained language models on M-patent evaluation dataset 70
Figure 4.5. Comparison of various threshold values for XLNet model on USPTO-2M dataset 73
Figure 4.6. Comparison of various threshold values for XLNet on M-patent dataset 74
Figure 4.7. Performance comparison of the models using different sections of the patent on the... 75
Figure 5.1. The overall structure of the Hi-Patnet model 86