CoLM-DSR: Leveraging Neural Codec Language Modeling for Multi-Modal Dysarthric Speech Reconstruction
1. Abstract
Dysarthric speech reconstruction (DSR) aims to transform dysarthric speech into normal speech. It still suffers from low speaker similarity and poor prosody naturalness. In this paper, we propose a multi-modal DSR model by leveraging neural codec language modeling to improve the reconstruction results, especially for the speaker similarity and prosody naturalness. Our proposed model consists of: (i) a multi-modal content encoder to extract robust phoneme embeddings from dysarthric speech with auxiliary visual inputs; (ii) a speaker codec encoder to extract and normalize the speaker-aware codecs from the dysarthric speech, in order to provide original timbre and normal prosody; (iii) a codec language model based speech decoder to reconstruct the speech based on the extracted phoneme embeddings and normalized codecs. Evaluations on the commonly used UASpeech corpus show that our proposed model can achieve significant improvements in terms of speaker similarity and prosody naturalness.
2. Proposed Model Architecture
3. Comparison with Different Baseline Systems
- AON-DSR: It uses an audio-only encoder to extract phoneme embeddings, a prosody corrector to explicitly model the duration and pitch, a speaker encoder to represent the speaker embedding, and a mel-decoder to reconstruct the melspectrogram based on the phoneme and prosody inputs.
- VGG-DSR: Following AON-DSR system, it uses a VGG-based audio-visual encoder instead of the audio-only encoder to extract phoneme embeddings.
- AVHu-DSR: Following AON-DSR system, it uses a AVHuBERT-based audio-visual encoder (similar to paper 2.1) instead of the audio-only encoder to extract phoneme embeddings.
- CoLM-DSR: Our proposed codec LM based multi-modal DSR system.
3.1.1 Text: Enter
| Original Video | Original Speech | ||
|---|---|---|---|
| AON-DSR | VGG-DSR | ||
| AVHu-DSR | CoLM-DSR |
3.1.2 Text: Shift
| Original Video | Original Speech | ||
|---|---|---|---|
| AON-DSR | VGG-DSR | ||
| AVHu-DSR | CoLM-DSR |
3.1.3 Text: Paragraph
| Original Video | Original Speech | ||
|---|---|---|---|
| AON-DSR | VGG-DSR | ||
| AVHu-DSR | CoLM-DSR |
3.1.4 Text: Word
| Original Video | Original Speech | ||
|---|---|---|---|
| AON-DSR | VGG-DSR | ||
| AVHu-DSR | CoLM-DSR |
3.1.5 Text: When
| Original Video | Original Speech | ||
|---|---|---|---|
| AON-DSR | VGG-DSR | ||
| AVHu-DSR | CoLM-DSR |
3.1.6 Text: One
| Original Video | Original Speech | ||
|---|---|---|---|
| AON-DSR | VGG-DSR | ||
| AVHu-DSR | CoLM-DSR |
3.1.7 Text: Seven
| Original Video | Original Speech | ||
|---|---|---|---|
| AON-DSR | VGG-DSR | ||
| AVHu-DSR | CoLM-DSR |
3.2.1 Text: Line
| Original Video | Original Speech | ||
|---|---|---|---|
| AON-DSR | VGG-DSR | ||
| AVHu-DSR | CoLM-DSR |
3.2.2 Text: Copy
| Original Video | Original Speech | ||
|---|---|---|---|
| AON-DSR | VGG-DSR | ||
| AVHu-DSR | CoLM-DSR |
3.2.3 Text: Downward
| Original Video | Original Speech | ||
|---|---|---|---|
| AON-DSR | VGG-DSR | ||
| AVHu-DSR | CoLM-DSR |
3.2.4 Text: Bravo
| Original Video | Original Speech | ||
|---|---|---|---|
| AON-DSR | VGG-DSR | ||
| AVHu-DSR | CoLM-DSR |
3.2.5 Text: Kilo
| Original Video | Original Speech | ||
|---|---|---|---|
| AON-DSR | VGG-DSR | ||
| AVHu-DSR | CoLM-DSR |
3.2.6 Text: Tango
| Original Video | Original Speech | ||
|---|---|---|---|
| AON-DSR | VGG-DSR | ||
| AVHu-DSR | CoLM-DSR |
4. Investigation on Dysarthric Codecs and Normal Codecs
- Dysarthric: We use the dysarthric codecs directly to generate the reconstructed speech.
- Normal: We use the converted normal codecs to generate the reconstructed speech.
4.1 Text: Although
| Original Video | Original Speech | ||
|---|---|---|---|
| Dysarthric | Normal |
4.2 Text: Bulrush
| Original Video | Original Speech | ||
|---|---|---|---|
| Dysarthric | Normal |
4.3 Text: Left
| Original Video | Original Speech | ||
|---|---|---|---|
| Dysarthric | Normal |
4.4 Text: People
| Original Video | Original Speech | ||
|---|---|---|---|
| Dysarthric | Normal |
4.5 Text: Lima
| Original Video | Original Speech | ||
|---|---|---|---|
| Dysarthric | Normal |
4.6 Text: Quebec
| Original Video | Original Speech | ||
|---|---|---|---|
| Dysarthric | Normal |