[ASRU 2023] Scenario-Aware Audio-Visual TF-GridNet for Target Speech Extraction

MERL Researcher Zexu Pan presents his paper titled "Scenario-Aware Audio-Visual TF-GridNet for Target Speech Extraction" for the IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), held 16-20 December 2023. The paper was co-authored with MERL researchers Gordon Wichern, Yoshiki Masuyama, Francois G. Germain, Sameer Khurana, Chiori Hori, and Jonathan Le Roux.
Abstract: Target speech extraction aims to extract, based on a given conditioning cue, a target speech signal that is corrupted by interfering sources, such as noise or competing speakers. Building upon the achievements of the state-of-the-art (SOTA) time-frequency speaker separation model TF-GridNet, we propose AV-GridNet, a visual-grounded variant that incorporates the face recording of a target speaker as a conditioning factor during the extraction process. Recognizing the inherent dissimilarities between speech and noise signals as interfering sources, we also propose SAV-GridNet, a scenario-aware model that identifies the type of interfering scenario first and then applies a dedicated expert model trained specifically for that scenario. Our proposed model achieves SOTA results on the second COG-MHEAR Audio-Visual Speech Enhancement Challenge, outperforming other models by a significant margin. We also perform an extensive analysis of the results under the two scenarios.