Xutai Ma

A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Tasks

2021

Attention-based sequence-to-sequence modeling provides a powerful and elegant solution for applications that need to map one sequence to a different sequence. Its success heavily relies on the availability of large amounts of training data. This …

Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Tasks

2023 · arXiv (Cornell University)

Transducer and Attention based Encoder-Decoder (AED) are two widely used frameworks for speech-to-text tasks. They are designed for different purposes and each has its own benefits and drawbacks for speech-to-text tasks. In order to leverage …

SSR: Alignment-Aware Modality Connector for Speech Language Models

2025

Fusing speech into a pre-trained language model (SpeechLM) usually suffers from the inefficient encoding of long-form speech and catastrophic forgetting of pre-trained text modality.We propose SSR-CONNECTOR (Segmented Speech Representation Connector) for better modality fusion.Leveraging speech-text …