AudioPaLM: A Large Language Model That Can Speak and Listen. (arXiv:2306.12925v1 [cs.CL])
By: <a href="http://arxiv.org/find/cs/1/au:+Rubenstein_P/0/1/0/all/0/1">Paul K. Rubenstein</a>, <a href="http://arxiv.org/find/cs/1/au:+Asawaroengchai_C/0/1/0/all/0/1">Chulayuth Asawaroengchai</a>, <a href="http://arxiv.org/find/cs/1/au:+Nguyen_D/0/1/0/all/0/1">Duc Dung Nguyen</a>, <a href="http://arxiv.org/find/cs/1/au:+Bapna_A/0/1/0/all/0/1">Ankur Bapna</a>, <a href="http://arxiv.org/find/cs/1/au:+Borsos_Z/0/1/0/all/0/1">Zalán Borsos</a>, <a href="http://arxiv.org/find/cs/1/au:+Quitry_F/0/1/0/all/0/1">Félix de Chaumont Quitry</a>, <a href="http://arxiv.org/find/cs/1/au:+Chen_P/0/1/0/all/0/1">Peter Chen</a>, <a href="http://arxiv.org/find/cs/1/au:+Badawy_D/0/1/0/all/0/1">Dalia El Badawy</a>, <a href="http://arxiv.org/find/cs/1/au:+Han_W/0/1/0/all/0/1">Wei Han</a>, <a href="http://arxiv.org/find/cs/1/au:+Kharitonov_E/0/1/0/all/0/1">Eugene Kharitonov</a>, <a href="http://arxiv.org/find/cs/1/au:+Muckenhirn_H/0/1/0/all/0/1">Hannah Muckenhirn</a>, <a href="http://arxiv.org/find/cs/1/au:+Padfield_D/0/1/0/all/0/1">Dirk Padfield</a>, <a href="http://arxiv.org/find/cs/1/au:+Qin_J/0/1/0/all/0/1">James Qin</a>, <a href="http://arxiv.org/find/cs/1/au:+Rozenberg_D/0/1/0/all/0/1">Danny Rozenberg</a>, <a href="http://arxiv.org/find/cs/1/au:+Sainath_T/0/1/0/all/0/1">Tara Sainath</a>, <a href="http://arxiv.org/find/cs/1/au:+Schalkwyk_J/0/1/0/all/0/1">Johan Schalkwyk</a>, <a href="http://arxiv.org/find/cs/1/au:+Sharifi_M/0/1/0/all/0/1">Matt Sharifi</a>, <a href="http://arxiv.org/find/cs/1/au:+Tadmor_M/0/1/0/all/0/1">Michelle Tadmor</a>, <a href="http://arxiv.org/find/cs/1/au:+Ramanovich/0/1/0/all/0/1">Ramanovich</a>, <a href="http://arxiv.org/find/cs/1/au:+Tagliasacchi_M/0/1/0/all/0/1">Marco Tagliasacchi</a>, <a href="http://arxiv.org/find/cs/1/au:+Tudor_A/0/1/0/all/0/1">Alexandru Tudor</a>, <a href="http://arxiv.org/find/cs/1/au:+Velimirovic_M/0/1/0/all/0/1">Mihajlo Velimirović</a>, <a href="http://arxiv.org/find/cs/1/au:+Vincent_D/0/1/0/all/0/1">Damien Vincent</a>, <a href="http://arxiv.org/find/cs/1/au:+Yu_J/0/1/0/all/0/1">Jiahui Yu</a>, <a href="http://arxiv.org/find/cs/1/au:+Wang_Y/0/1/0/all/0/1">Yongqiang Wang</a>, <a href="http://arxiv.org/find/cs/1/au:+Zayats_V/0/1/0/all/0/1">Vicky Zayats</a>, <a href="http://arxiv.org/find/cs/1/au:+Zeghidour_N/0/1/0/all/0/1">Neil Zeghidour</a>, <a href="http://arxiv.org/find/cs/1/au:+Zhang_Y/0/1/0/all/0/1">Yu Zhang</a>, <a href="http://arxiv.org/find/cs/1/au:+Zhang_Z/0/1/0/all/0/1">Zhishuai Zhang</a>, <a href="http://arxiv.org/find/cs/1/au:+Zilka_L/0/1/0/all/0/1">Lukas Zilka</a>, <a href="http://arxiv.org/find/cs/1/au:+Frank_C/0/1/0/all/0/1">Christian Frank</a> Posted: June 23, 2023
We introduce AudioPaLM, a large language model for speech understanding and
generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2
[Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified
multimodal architecture that can process and generate text and speech with
applications including speech recognition and speech-to-speech translation.
AudioPaLM inherits the capability to preserve paralinguistic information such
as speaker identity and intonation from AudioLM and the linguistic knowledge
present only in text large language models such as PaLM-2. We demonstrate that
initializing AudioPaLM with the weights of a text-only large language model
improves speech processing, successfully leveraging the larger quantity of text
training data used in pretraining to assist with the speech tasks. The
resulting model significantly outperforms existing systems for speech
translation tasks and has the ability to perform zero-shot speech-to-text
translation for many languages for which input/target language combinations
were not seen in training. AudioPaLM also demonstrates features of audio
language models, such as transferring a voice across languages based on a short
spoken prompt. We release examples of our method at
https://google-research.github.io/seanet/audiopalm/examples
Provided by:
http://arxiv.org/icons/sfx.gif
