Skip to main navigation Skip to search Skip to main content

An efficient minimum vocabulary construction algorithm for language modeling

  • Beihang University
  • Boston University

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

In learning a new word by a dictionary, we first need to know a set of "basic words" which are frequently appeared in word definitions. It often happens that you cannot understand the word you looked up because there are still some words you do not understand in its definitions or explanations provided by the dictionary. You can keep looking up these new words recursively till they all can be well explained by some basic words you already knew. How to automatically find a minimum set of such basic words to define (or recursively define) the entire vocabulary in a given dictionary is what are going to discuss in this paper. We propose an efficient algorithm to construct the Minimum Vocabulary (MV) using the word frequency information. The minimum vocabulary can be used for language modeling and experimental results demonstrate the effectiveness of using the minimum vocabulary as features in text classification.

Original languageEnglish
Title of host publicationAdvanced Research in Applied Artificial Intelligence - 25th International Conference on Industrial Engineering and Other Applications of Applied Intelligent Systems, IEA/AIE 2012, Proceedings
Pages292-301
Number of pages10
DOIs
StatePublished - 2012
Event25th International Conference on Industrial Engineering and Other Applications of Applied Intelligent Systems, IEA/AIE 2012 - Dalian, China
Duration: 9 Jun 201212 Jun 2012

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume7345 LNAI
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference25th International Conference on Industrial Engineering and Other Applications of Applied Intelligent Systems, IEA/AIE 2012
Country/TerritoryChina
CityDalian
Period9/06/1212/06/12

Fingerprint

Dive into the research topics of 'An efficient minimum vocabulary construction algorithm for language modeling'. Together they form a unique fingerprint.

Cite this