Hybrid Approaches to Device-Directed Speech Detection Using Multimodal Data

Device-directed speech detection (DDSD) is essential for any speech-based interface and simplifies the utilization of voice-activated systems, especially in areas with high levels of noise interference. This paper studies the enhancement of DDSD (Distributed Denial of Service Detection) with multimodal fusion techniques. These approaches include the combination of the audio and the video feed to increase the detection level. The models were established to predict the baseline, where only the audio or video characteristics were incorporated with an accuracy of 75. 4% and 77. This, in turn, means that their recognition capability is at 8% correspondingly and highlights their limitations in any given environment with high noise levels. When combining audio and video information at the feature level, the accuracy was raised to 85—6% when the data was combined into a unified vector. Freescale also improved its performance by up to 88 percent. , decision-level fusion by achieving an accuracy of 2% from the fused classifiers' decision for each modality. The feature-level fusion technique and decision-level fusion produced the highest accuracy of 92% for the given methodology. The hybrid fusion model also presented outstanding AOC formative accuracy, recall, and an F1 score of 92. 8%, 92. 1%, and 92. Specifically, its performance in moderate noise levels, low noise levels, and interfering voice conditions was 87%, 89%, and 4%, respectively, thereby revealing its ability to cope with diverse noisy environments. The examination of the hybrid model's ROC curve depicted the model's excellent capacity in discriminating between the target customers, with an AUC of 0. 96%.