Title Page
ABSTRACT
Contents
1. Introduction 18
1.1. Research Background 18
1.2. Definition of Multimodality and Multimodal System 21
1.3. Research Rationale 24
1.4. Research Questions 25
1.5. Outline of Research 28
2. Literature Review 30
2.1. Modality and Multimodality 30
2.2. Multimodal Interface and System 33
2.3. Effects of Multimodal System 37
2.4. Strengths of Multimodal System 41
2.5. Cognitive Patterns Related to Multimodal Interaction 43
2.5.1. Serial Processing 43
2.5.2. Parallel processing 44
2.5.3. Cognitive Patterns of Multimodal Interaction 47
2.6. Related Works 50
3. Method 55
3.1. Mode and Task Definitions 56
3.2. Subjects 59
3.3. Experimental Design 60
3.4. Apparatus 62
3.5. Experimental Procedure 64
4. Results of Multimodal Effect for the Simple and Complex Tasks 65
4.1. Multimodal Effect for the Simple Task 65
4.1.1. Multimodal effect of the Sequential Input Mode 65
4.1.2. Multimodal Effect for the Simultaneous Input Mode 72
4.1.3. Discussion 75
4.2. Multimodal Effect for the Complex Task 77
4.2.1. Multimodal Effect for the Sequential Input Mode 77
4.2.2. Multimodal Effect for the Simultaneous Input Mode 90
4.2.3. Discussion 94
4.3. Summary of Multimodal Effects for the Simple and Complex Tasks 97
4.3.1. Results of Multimodal Effect for the Simple and Complex Task 97
4.3.2. Discussion 100
5. Multimodal Effect with the Finger Gesture Input Mode 102
5.1. Method 102
5.1.1. Subjects and Apparatus 104
5.1.2. Experimental Design and Procedure 106
5.2. Results of Multimodal Effect of Sequential Input Mode 108
5.3. Results of Multimodal Effect of Simultaneous Input Mode 113
5.4. Discussion 115
6. General Discussion 122
7. Conclusions 129
References 132
Table 2-1. The category of the mode or modality, taken from Karray, 2008 31
Table 2-2. Ten myths of multimodal interaction, taken from Oviatt, 1999b 37
Table 2-3. Effects of multimodal interaction with quality factors 54
Table 3-1. Sequential and simultaneous input modes in multimodal interaction experiments 57
Table 3-2. Tasks in the first and second experiments 58
Table 4-1. ANOVA results of the experiment with respect to the user group and mode 66
Tabl3 4-2. ANOVA results of the experiment with respect to each user group and mode 68
Table 4-3. Error rate of modes in each user group 72
Table 4-4. Error rates of mode selections in each user group 75
Table 4-5. ANOVA results of the experiment with respect to the user groups, tasks and modes 78
Table 4-6. ANOVA results of the experiment with respect to each user group, task and mode 85
Table 4-7. Error rate of the task and mode in each user group 89
Table 4-8. Error rate of mode selection in each user group 93
Table 4-9. Comparison of preference scores and performances among user and system factors 98
Table 5-1. Sequential and simultaneous input modes for the multimodal interaction experiments 103
Table 5-2. Tasks in the first and second experiments 104
Table 5-3. ANOVA results of the first experiment with respect to three tasks and nine input modes 108
Table 5-4. Error rates in terms of error frequency for the tasks and sequential input modes 110
Table 5-5. Error rates in terms of frequency for task and simultaneous input mode selection 115
Table 5-6. Comparison of preference and performance between the sequential and simultaneous... 119
Figure 1-1. Quality factors of multimodal interaction, taken from Wechsung, et al., 2012 20
Figure 1-2. Quality factors for multimodal interaction on a smart TV 28
Figure 2-1. The multimodal interface and system 33
Figure 2-2. The example of the multimodal interface and system development 35
Figure 3-1. Multimodal system of a smart TV used in multimodal interaction experiments 62
Figure 4-1. Mean of the interaction effect between the user group and mode 67
Figure 4-2. Mean preference score and completion time for sequential input modes in group 1 70
Figure 4-3. Mean preference score and completion time for sequential input modes in group 2 71
Figure 4-4. Percentages of mode selection for simultaneous input modes 73
Figure 4-5. Mean completion time in seconds for sequential and simultaneous input modes 74
Figure 4-6. Mean of the interaction effect between the user group and mode 79
Figure 4-7. Mean interaction effect between task and mode 82
Figure 4-8. Comparison of preference scores and completion times between V-M and M-V... 84
Figure 4-9. Mean preference and completion times for sequential input modes in group 1 87
Figure 4-10. Mean preference and completion times for sequential input modes in group 2 88
Figure 4-11. Proportions of selection of simultaneous input modes 91
Figure 4-12. Mean completion time in seconds for sequential and simultaneous input modes 93
Figure 5-1. Multimodal system for a smart TV in multimodal interaction experiments 105
Figure 5-2. Mean preferences in terms of rating score for sequential input modes 109
Figure 5-3. Mean completion times in second for sequential input modes 110
Figure 5-4. Mean values of interaction effects between task and mode 112
Figure 5-5. Proportion of mode selection frequency per task in percentage 114
Figure 5-6. Mean completion times in second for simultaneous input modes 114
Figure 5-7. Comparison of preferences between unimodal and multimodal input modes 117