Clustering for data compression

Invention relates to profileometry in the field of non-destructive measurement of wafer layers in the basic wafer-fabrication operations of layering, patterning, doping and heat treatments. An improved profileometry data collection and analysis system employing software that performs clustering analysis on library data stored in memory that represent semiconductor chip wafer profiles, for use in matching real-time data signals from data collected by profileometry instruments. To better perform a match in real-time between the incoming real-time data signals and the profile library data, cluster analysis is performed on the library data to partition the library data into clusters, and to extract representative cluster data points of the clusters. The representatives of the clusters are stored in primary memory (e.g., RAM), while the data forming the clusters are stored in secondary memory (e.g., a hard drive). A real-time data signal is then first compared to the representative cluster data points, and when a match is made with a particular representative cluster data point, the cluster associated with the representative cluster data point is loaded from secondary memory into primary memory. Next a further search is made with the incoming real-time data signal to find the closest match to the data in the cluster. In this way the entire library data does not have to be searched sequentially, and the entire library does not have to reside in primary memory in order to be quickly searched, which both conserves time and primary memory.