Data quality evaluation and improvement for prognostic modeling using visual assessment based data partitioning method

► Data quality evaluation and improvement facilitate the design of prognostics modeling. ► Data quality evaluation procedures with regard to modeling of system health diagnosis are defined. ► Visual assessment based data partitioning. ► Outlier detection based on visual analysis of data spectrum. When developing Prognostic and Health Management (PHM) applications for manufacturing systems, data acquired frequently comes with issues which hinder further data analysis. However, there is neither a clear definition of the data quality nor evaluation methods to quantify if acquired data is suitable for these prognostic modeling tasks such as failures detection, diagnosis and prediction. Especially, during health diagnosis modeling of engineering systems, based on data-driven method, acquired data is expected to contain clusters that can be used to differentiate multiple system health conditions. So in most cases, once data is acquired, people would like to intuitively believe that data is able to cluster into subgroups. However, this bias could lead to acceptance of false information in data. Furthermore, most of the existing metrics, such as clustering tendency in statistics and cluster-ability in data mining, only individually evaluate data characteristics without considering prognostic modeling. This paper proposes a new method to evaluate and improve data quality for system health diagnosis modeling. The clusters, as critical data characteristics for modeling multiple system conditions, are first estimated by “visualization” on the dissimilarity spectrum from spectral analysis and then evaluated in terms of their fitness and separation with each others. A visual assessment based outlier detection method is also proposed to recognize outliers from the data, which utilizes the graphic intermediate results from previous evaluation. Finally one group of bearing testing dataset acquired from real industrial applications is used to demonstrate how proposed methods are used to evaluate and improve the data quality.