Effective Clustering of Text Documents in Low Dimension Space Using Semantic Association Among Terms

Abstract

Sparse and high dimensional document representation of the popular Vector Space Model results in poor clustering performance. Dimension reduction techniques are useful for dense and low dimensional representation of documents that enhances clustering performance. This paper proposes a novel unsupervised filter method for feature selection. Filter methods assign weights to terms, used for representation of documents in the collection, according to some criterion, which is different from clustering task. Unsupervised feature selection methods do not use class labels to guide the selection of features. The proposed method assigns a score to a term, which is proportional to the term’s overall semantic association with rest of the terms in the document collection. The overall semantic association of a term is estimated using the co-occurrence frequencies of the term with other terms in the collection. Clustering results on three ideal text data sets TDT2, Reuters21578 and 20 Newsgroups proved that the proposed method selects features that are more discriminative, to separate intrinsic classes of documents, when compared with that selected by the existing unsupervised filter based feature section methods

Copyright © 2015 Praise Worthy Prize - All rights reserved.