Audio-visual End-to-end Multi-channel Speech Separation, Dereverberation and Recognition
Publication in refereed journal
Officially Accepted for Publication


Full Text

Other information
AbstractAccurate recognition of cocktail party speech containing overlapping speakers, noise and reverberation remains a highly challenging task to date. Motivated by the invariance of visual modality to acoustic signal corruption, an audio-visual multi-channel speech separation, dereverberation and recognition approach featuring a full incorporation of visual information into all system components is proposed in this paper. The efficacy of the video input is consistently demonstrated in mask- based MVDR speech separation, DNN-WPE or spectral mapping (SpecM) based speech dereverberation front-end and Conformer ASR back-end. Audio-visual integrated front-end architectures performing speech separation and dereverberation in a pipelined or joint fashion via mask-based WPD are investigated. The error cost mismatch between the speech enhancement front-end and ASR back-end components is minimized by end-to-end jointly fine-tuning using either the ASR cost function alone, or its interpolation with the speech enhancement loss. Experiments were conducted on the mixture overlapped and reverberant speech data constructed using simulation or replay of the Oxford LRS2 dataset. The proposed audio-visual multi-channel speech separation, dereverberation and recognition systems consistently outperformed the comparable audio-only baseline by 9.1% and 6.2% absolute (41.7% and 36.0% relative) word error rate (WER) reductions. Consistent speech enhancement improve- ments were also obtained on PESQ, STOI and SRMR scores.
Acceptance Date30/06/2023
All Author(s) ListGuinan Li, Jiajun Deng, Mengzhe Geng, Zengrui Jin, Tianzi Wang, Shujie Hu, Mingyu Cui, Helen Meng, Xunying Liu
Journal nameIEEE/ACM Transactions on Audio, Speech and Language Processing
Year2023
ISSN2329-9290
LanguagesEnglish-United States
KeywordsAudio-visual, Speech separation, Speech dereverberation, Speech recognition, End-to-end, Conformer

Last updated on 2023-31-07 at 16:51