DiffDSR: Dysarthric Speech Reconstruction Using Latent Diffusion Model

(Accepted by Interspeech 2025)

1. Abstract

Dysarthric speech reconstruction (DSR) aims to convert dysarthric speech into comprehensible speech while maintaining the speaker’s identity. Despite significant advancements, existing methods often struggle with low speech intelligibility and poor speaker similarity. In this study, we introduce a novel diffusion-based DSR system that leverages a latent diffusion model to enhance the quality of speech reconstruction. Our model comprises: (i) a speech content encoder for phoneme embedding restoration via pre-trained self-supervised learning (SSL) speech foundation models; (ii) a speaker identity encoder for speaker-aware identity preservation by in-context learning mechanism; (iii) a diffusion-based speech generator to reconstruct the speech based on the restored phoneme embedding and preserved speaker identity. Through evaluations on the widely-used UASpeech corpus, our proposed model shows notable enhancements in speech intelligibility and speaker similarity.

2. Proposed Model Architecture

3. Comparison with Different Baseline Systems

• SV-DSR: It uses a speaker encoder to extract a global timbre embedding and a multi-speaker mel-based decoder.
• CoLM-DSR: Excluding the influence of multi-modal input, it uses a LM-based generator with speech codec prompt.
• Diff-DSR: Our complete proposed diffusion based system.

3.1 Speaker: M12

3.1.1 Text: Left

Original	SV-DSR	CoLM-DSR	Diff-DSR

3.1.2 Text: Juliet

Original	SV-DSR	CoLM-DSR	Diff-DSR

3.1.3 Text: Whiskey

Original	SV-DSR	CoLM-DSR	Diff-DSR

3.1.4 Text: Many

Original	SV-DSR	CoLM-DSR	Diff-DSR

3.1.5 Text: Golf

Original	SV-DSR	CoLM-DSR	Diff-DSR

3.1.6 Text: Watch

Original	SV-DSR	CoLM-DSR	Diff-DSR

3.2 Speaker: F02

3.2.1 Text: Paragraph

Original	SV-DSR	CoLM-DSR	Diff-DSR

3.2.2 Text: Word

Original	SV-DSR	CoLM-DSR	Diff-DSR

3.2.3 Text: When

Original	SV-DSR	CoLM-DSR	Diff-DSR

3.2.4 Text: Seven

Original	SV-DSR	CoLM-DSR	Diff-DSR

3.2.5 Text: Foxtrot

Original	SV-DSR	CoLM-DSR	Diff-DSR

3.3 Speaker: M16

3.3.1 Text: Copy

Original	SV-DSR	CoLM-DSR	Diff-DSR

3.3.2 Text: Bravo

Original	SV-DSR	CoLM-DSR	Diff-DSR

3.3.3 Text: Kilo

Original	SV-DSR	CoLM-DSR	Diff-DSR

3.3.4 Text: Oscar

Original	SV-DSR	CoLM-DSR	Diff-DSR

3.3.5 Text: Tango

Original	SV-DSR	CoLM-DSR	Diff-DSR

3.3.6 Text: Upward

Original	SV-DSR	CoLM-DSR	Diff-DSR

3.4 Speaker: F04

3.4.1 Text: Bulrush

Original	SV-DSR	CoLM-DSR	Diff-DSR

3.4.2 Text: Juliet

Original	SV-DSR	CoLM-DSR	Diff-DSR

3.4.3 Text: Quebec

Original	SV-DSR	CoLM-DSR	Diff-DSR

3.4.4 Text: Uniform

Original	SV-DSR	CoLM-DSR	Diff-DSR

3.4.5 Text: Victor

Original	SV-DSR	CoLM-DSR	Diff-DSR