Latent Semantic Analysis for Multimodal User Input With Speech and Gestures
Publication in refereed journal

Times Cited
Web of Science7WOS source URL (as at 22/02/2021) Click here for the latest count
Altmetrics Information

Other information
AbstractThis paper describes our work in semantic interpretation of a "multimodal language" with speech and gestures using latent semantic analysis (LSA). Our aim is to infer the domain-specific informational goal of multimodal inputs. The informational goal is characterized by lexical terms used in the spoken modality, partial semantics of gestures in the pen modality, as well as term co-occurrence patterns across modalities, leading to " multimodal terms." We designed and collected a multimodal corpus of navigational inquiries. We also obtained perfect (i.e. manual) and imperfect (i.e. automatic via recognition) transcriptions for these. We automatically align parsed spoken locative references (SLRs) with their corresponding pen gesture(s) using the Viterbi alignment, according to their numeric and location type features. Then, we characterize each cross-modal integration pattern as a 3-tuple multimodal term with SLR, pen gesture type and their temporal relationship. We propose to use latent semantic analysis (LSA) to derive the latent semantics from manual (i.e. perfect) and automatic (i.e. imperfect) transcriptions of the collected multimodal inputs. In order to achieve this, both multimodal and lexical terms are used to compose an inquiry-term matrix, which is then factorized using singular value decomposition (SVD) to derive the latent semantics automatically. Informational goal inference based on the latent semantics shows that the informational goal inference accuracy of a disjoint test set is 99% and 84% when a perfect and imperfect projection model is used respectively, which performs significantly better than (at least 9.9% absolute) the baseline performance using vector-space model (VSM).
All Author(s) ListHui PY, Meng HL
Journal nameIEEE/ACM Transactions on Audio, Speech and Language Processing
Volume Number22
Issue Number2
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Pages417 - 429
LanguagesEnglish-United Kingdom
Keywordsgesture recognition; latent semantic analysis.; Multimodal user interfaces; speech recognition
Web of Science Subject CategoriesAcoustics; Engineering; Engineering, Electrical & Electronic

Last updated on 2021-23-02 at 00:12