Weiting Tan

SSR: Alignment-Aware Modality Connector for Speech Language Models

2025

Fusing speech into a pre-trained language model (SpeechLM) usually suffers from the inefficient encoding of long-form speech and catastrophic forgetting of pre-trained text modality.We propose SSR-CONNECTOR (Segmented Speech Representation Connector) for better modality fusion.Leveraging speech-text …

Process-Supervised Reinforcement Learning for Interactive Multimodal Tool-Use Agents

2025 · arXiv (Cornell University)

Effective interactive tool use requires agents to master Tool Integrated Reasoning (TIR): a complex process involving multi-turn planning and long-context dialogue management. To train agents for this dynamic process, particularly in multi-modal contexts, we introduce …