Project
Improvements of denoising language models for speech recognition
Automatic speech recognition (ASR) typically relies on an acoustic model (AM) to map audio features to text sequences, alongside a language model (LM) to estimate word distributions over text corpora. While supervised audio-text pairs are resource-constrained and costly to produce, unstructured text-only data is widely available in massive volumes. Traditional language models integrate this text data via autoregressive structures or decoding fusion methods, but they lack full bi-directional context and are not explicitly optimized for typical phonetic or acoustic error patterns generated by the base AM. Denoising language models (DLMs) solve this by acting as error correction layers that map corrupted text hypotheses back to true underlying transcriptions. Using the LibriSpeech benchmark, this project systematically evaluates multi-stage decoding techniques and data perturbation strategies to maximize text-only utility and push correction boundaries.
Project Details
Project term
December 5, 2024–March 4, 2026
Affiliations
RWTH Aachen University
Institute
Chair of Machine Learning and Reasoning
Principal Investigator
Methods
The experimental framework utilizes a Text-to-Speech (TTS) synthesis system trained on LibriSpeech to convert an 800-million-word text corpus into roughly 75,000 hours of synthetic speech. Passing this audio through a base acoustic model yields error-prone text hypotheses. The DLM is subsequently trained on these aligned pairs of noisy hypotheses and clean ground-truth text using a cross-entropy loss objective. The core architecture consists of a 471M parameter CTC Conformer AM and a 466M parameter Transformer attention-based encoder-decoder DLM. To optimize the correction pipeline, we introduce advanced search algorithms—specifically Denoising Speech Recognition (DSR) and DLM-sum decoding—which are benchmarked against text-level training data perturbations including SpecAugment time/frequency masking, dropout variations, token substitution, and mixup regularization.
Results
On the LibriSpeech test-other dataset, the baseline ASR model achieves a greedy word error rate (WER) of 4.44%, while a standard 422M parameter LM with one-pass decoding reaches 3.70%. Passing the hypotheses through the DLM using simple greedy decoding degrades performance to 4.57% due to text hallucinations. However, DSR decoding recovers this to 3.70%, and DLM-sum decoding achieves the optimal overall WER of 3.53%.
Ablation benchmarks confirm that data augmentation is vital. Compared to the unaugmented baseline at 4.00% WER, the low+ configuration yields the best performance at 3.60%. Word-level tracking shows that DLM-sum maximizes true corrections while minimizing introduced errors.
Discussion
The experiments verify that DLMs using global context provide superior error correction over traditional causal LMs, provided they use advanced search techniques. Simple greedy decoding is highly vulnerable to ungrounded loops and hallucinations. Incorporating DSR or DLM-sum decoding resolves this by enforcing acoustic checks and marginalizing search paths over multiple ASR hypotheses.
Analysis indicates that synthetic TTS training data is essential, but it requires balanced perturbation via intermediate check-pointing or masking to generalize well. Categorically, the DLM shows uniform utility across parts of speech but remains constrained on rare words. Future work should replace the heavy TTS stage with a text noising model.
Additional Project Information
DFG classification: 409-05 Interactive and Intelligent Systems, Image and Language Processing, Computer Graphics and Visualisation
Cluster: CLAIX
Publications
Dorian Koch, Albert Zeyer, Nick Rossenbach, Ralf Schlüter, Hermann Ney,
Reproducing and Dissecting Denoising Language Models for Speech Recognition,
https://dx.doi.org/10.48550/arXiv.2512.13576, December 2025
Albert Zeyer, Tim Posielek, Ralf Schlüter, Hermann Ney,
Text-Utilization for Encoder-dominated Speech Recognition Models,
https://dx.doi.org/10.48550/arXiv.2604.26514, April 2026
Davyd Naveriani, Albert Zeyer, Ralf Schlüter, Hermann Ney,
Diffusion Language Models for Speech Recognition,
https://dx.doi.org/10.48550/arXiv.2604.14001, April 2026
Thesis:
Dorian Koch,
Investigating Denoising Language Models for Automatic Speech Recognition,
Bachelor thesis, October 2025