37 min listen
StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning
StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning
ratings:
Length:
41 minutes
Released:
Jun 20, 2024
Format:
Podcast episode
Description
Simultaneous speech-to-speech translation (Simul-S2ST, a.k.a streaming speech translation) outputs target speech while receiving streaming speech inputs, which is critical for real-time communication. Beyond accomplishing translation between speech, Simul-S2ST requires a policy to control the model to generate corresponding target speech at the opportune moment within speech inputs, thereby posing a double challenge of translation and policy. In this paper, we propose StreamSpeech, a direct Simul-S2ST model that jointly learns translation and simultaneous policy in a unified framework of multi-task learning. Adhering to a multi-task learning approach, StreamSpeech can perform offline and simultaneous speech recognition, speech translation and speech synthesis via an"All-in-One"seamless model. Experiments on CVSS benchmark demonstrate that StreamSpeech achieves state-of-the-art performance in both offline S2ST and Simul-S2ST tasks. Besides, StreamSpeech is able to present high-quality intermediate results (i.e., ASR or translation results) during simultaneous translation process, offering a more comprehensive real-time communication experience.
2024: Shaolei Zhang, Qingkai Fang, Shoutao Guo, Zhengrui Ma, Min Zhang, Yang Feng
https://arxiv.org/pdf/2406.03049
2024: Shaolei Zhang, Qingkai Fang, Shoutao Guo, Zhengrui Ma, Min Zhang, Yang Feng
https://arxiv.org/pdf/2406.03049
Released:
Jun 20, 2024
Format:
Podcast episode
Titles in the series (100)
MentalLLaMA: Interpretable Mental Health Analysis on Social Media with Large Language Models: With the development of web technology, social media texts are becoming a rich source for automatic mental health analysis. As traditional discriminative methods bear the problem of low interpretability, the recent large language models have been exp... by Papers Read on AI