Performance assessment of kernel density clustering for gene expression profile data

Kernel density smoothing techniques have been used in classification or supervised learning of gene expression profile (GEP) data, but their applications to clustering or unsupervised learning of those data have not been explored and assessed. Here we report a kernel density clustering method for analysing GEP data and compare its performance with the three most widely‐used clustering methods: hierarchical clustering, K‐means clustering, and multivariate mixture model‐based clustering. Using several methods to measure agreement, between‐cluster isolation, and within‐cluster coherence, such as the Adjusted Rand Index, the Pseudo F test, the r2 test, and the profile plot, we have assessed the effectiveness of kernel density clustering for recovering clusters, and its robustness against noise on clustering both simulated and real GEP data. Our results show that the kernel density clustering method has excellent performance in recovering clusters from simulated data and in grouping large real expression profile data sets into compact and well‐isolated clusters, and that it is the most robust clustering method for analysing noisy expression profile data compared to the other three methods assessed. Copyright © 2003 John Wiley & Sons, Ltd.