Implementation of Hearing-Loss System Using Multi-Model Neural Networks on AR Glasses

People with hearing impairments commonly face dual challenges in speech comprehension and environmental awareness. Traditional hearing aids primarily focus on sound amplification and compensation, resulting in limited functionality, underscoring the need for integrated solutions. In this study, we implement a Whisper-based Spatial Audio Recognition system on AR glasses (WSAR), an AR solution that simultaneously enhances communication efficiency and environmental safety. WSAR employs a microphone array to capture audio for sound-source localization. The speech signals are denoised using the DeepFilterNet model and then recognized by the Whisper model. In addition, the audio is fed into an Audio Spectrogram Transformer (AST) to identify hazardous or emergency sounds in the environment. The integrated information, including speech recognition results, environmental sound alerts, and sound source directions, is visualized on low-occlusion AR glasses, enabling users with hearing impairments to understand conversations better. Experimental results show that the system can efficiently recognize speech and warning sounds. Therefore, the WSAR system provides an effective solution for improving both communication efficiency and environmental safety for individuals with hearing impairments.