Dirichlet class language models for speech recognition

Jen-Tzung Chien*, Chuang Hua Chueh

*Corresponding author for this work

Research output: Contribution to journalArticle

50 Scopus citations


Latent Dirichlet allocation (LDA) was successfully developed for document modeling due to its generalization to unseen documents through the latent topic modeling. LDA calculates the probability of a document based on the bag-of-words scheme without considering the order of words. Accordingly, LDA cannot be directly adopted to predict words in speech recognition systems. This work presents a new Dirichlet class language model (DCLM), which projects the sequence of history words onto a latent class space and calculates a marginal likelihood over the uncertainties of classes, which are expressed by Dirichlet priors. A Bayesian class-based language model is established and a variational Bayesian procedure is presented for estimating DCLM parameters. Furthermore, the long-distance class information is continuously updated using the large-span history words and is dynamically incorporated into class mixtures for a cache DCLM. Different language models are experimentally evaluated using the Wall Street Journal (WSJ) corpus. The amount of training data and the size of vocabulary are evaluated. We find that the cache DCLM effectively characterizes the unseen n-gram events and stores the class information for long-distance language modeling. This approach outperforms the other class-based and topic-based language models in terms of perplexity and recognition accuracy. The DCLM and cache DCLM achieved relative gain of word error rate by 3% to 5% over the LDA topic-based language model with different sizes of training data.

Original languageEnglish
Article number5466224
Pages (from-to)482-495
Number of pages14
JournalIEEE Transactions on Audio, Speech and Language Processing
Issue number3
StatePublished - 1 Jan 2011


  • Bayes procedure
  • clustering method
  • natural language
  • smoothing method
  • speech recognition

Fingerprint Dive into the research topics of 'Dirichlet class language models for speech recognition'. Together they form a unique fingerprint.

  • Cite this