Semantic, Hierarchical, Online Clustering of Web Search Results

We propose a Semantic, Hierarchical, Online Clustering (SHOC) approach to automatically organizing Web search results into groups. SHOC combines the power of two novel techniques, key phrase discovery and orthogonal clustering, to generate clusters which are both reasonable and readable. Moreover, SHOC can work for multiple languages: not only English but also oriental languages like Chinese. The main contribution of this paper includes the following. (1) The benefits of using key phrases as Web document features are discussed. A key phrase discovery algorithm based on suffix array is presented. This algorithm is highly effective and efficient no matter how large the language’s alphabet is. (2) The concept of orthogonal clustering is proposed for general clustering problems. The reason why matrix Singular Value Decomposition (SVD) can provide solution to orthogonal clustering is strictly proved. The orthogonal clustering has a solid mathematics foundation and many advantages over traditional heuristic clustering algorithms.