Online Monitoring of Heterogeneous Partially Observable Data Streams Based on Q-Learning

With the rapid advances in Internet of Things (IoT) technology and computational infrastructure, heterogeneous data streams are becoming common in various manufacturing applications. Meanwhile, the resource constraints often restrict the full observability of data streams due to limited budget for deploying or turning on every sensor at a time, as well as limited transmission and processing time for collecting high frequency information from all data streams, which poses significant challenges for multivariate statistical process control (SPC) and quality improvement. In this article, diverging from conventional heuristic approaches, we propose a new algorithm based on Q-learning to online monitor and quickly detect mean shifts occurring to heterogeneous data streams in the context of limited resources, where only a subset of observations is available at each acquisition time. In particular, we integrate Q-learning with a nonparametric cumulative sum (CUSUM) procedure to effectively detect a wide range of possible mean shifts when data streams follow arbitrary distributions. Both simulations and a case study are thoroughly conducted to evaluate the performance and demonstrate the superiority of the proposed method. Note to Practitioners -This paper is motivated by the practical issue of online process monitoring and anomaly detection with resource limitations. In particular, we can only select a subset of data streams to monitor at each time epoch, and the challenges are to dynamically choose which ones to observe and when to raise an alarm. Unlike the existing methodologies which are heuristic and only consider short-term rewards from the dynamic sampling, this paper proposes a novel monitoring and sampling strategy that allows the practitioners to cost-effectively monitor heterogeneous data streams by considering the long-term rewards. Three main steps are involved in the proposed method: (i) construct a local nonparametric monitoring statistic for each data stream; (ii) train the proposed reinforcement learning framework; and (iii) at each time epoch, determine the most informative data stream to observe according to the Q-table and decide whether to raise the alarm. Experimental results through simulations and a case study have shown that the proposed method has better performance than the existing methods in reducing detection delay.