Self-Starting Monitoring and Dynamic Sampling of High-Dimensional Data Streams

In today's manufacturing industries, the development of sensor technology and Internet of Things has made real-time process monitoring of high-dimensional data increasingly vital. However, resource constraints, such as limited power, budget, and transmission capacity, often prevent access to full data streams in real time. This means that practitioners need to effectively monitor the process based on only partially observed data by dynamically deciding the sampling layout in real time. Another common challenge of process monitoring in practice is the lack of historical reference data, which can occur due to process/system upgrades or equipment replacements. To address these critical challenges, this paper proposes MASS (Monitoring with Adaptive Sampling under Self-starting scheme), a novel self-starting monitoring approach tailored to monitor high-dimensional data streams when only limited resources and historical reference data are available. Our monitoring framework is based on a quantile-based nonparametric CUSUM procedure with likelihood ratio-based statistics, and then the Thompson Sampling (TS) algorithm is adopted to handle partially observed data in the self-starting scenario. A key feature of our proposed method is its adaptive estimation of the out-of-control distribution and data quantiles, which ensures robust detection for various shifts in data streams with arbitrary and heterogeneous distributions, even in cases with limited reference data. The outperformance of the proposed method is demonstrated through simulation experiments and a real-world case study. Note to Practitioners -This paper is motivated by the critical challenges of online process monitoring when only limited resources (e.g., limited power availability, limited number of sensors, and limited transmission capacity) and limited historical in-control reference data are available. For example, consider a scenario where a newly established production system requires online monitoring across multiple data streams. In such cases, there is often a deficiency of reference data crucial for constructing a reliable control chart. Additionally, due to resource constraints, it is frequently infeasible to gather information from all streams associated with the production line at each epoch in real time. Unlike previous methods which require either a sufficient amount of reference data, or fully observable data streams, this paper proposes a novel monitoring and dynamic sampling scheme to ef