Soft detection of 5-day BOD with sparse matrix in city harbor water using deep learning techniques

To better control and manage harbor water quality is an important mission for coastal cities such as New York City (NYC). To achieve this, managers and governors need keep track of key quality indicators, such as temperature, pH, and dissolved oxygen. Among these, the Biochemical Oxygen Demand (BOD) over five days is a critical indicator that requires much time and effort to detect, causing great inconvenience in both academia and industry. Existing experimental and statistical methods cannot effectively solve the detection time problem or provide limited accuracy. Also, due to various human-made mistakes or facility issues, the data used for BOD detection and prediction contain many missing values, resulting in a sparse matrix. Few studies have addressed the sparse matrix problem while developing statistical detection methods. To address these gaps, we propose a deep learning based model that combines Deep Matrix Factorization (DMF) and Deep Neural Network (DNN). The model was able to solve the sparse matrix problem more intelligently and predict the BOD value more accurately. To test its effectiveness, we conducted a case study on the NYC harbor water, based on 32,323 water samples. The results showed that the proposed method achieved 11.54%–17.23% lower RMSE than conventional matrix completion methods, and 19.20%–25.16% lower RMSE than traditional machine learning algorithms. •This study proposed deep learning based techniques to softly detect 5-day BOD.•The sparse matrix problem is addressed using Deep Matrix Factorization (DMF).•The DMF part achieved 11.5%-17.2% lower RMSE than conventional methods.•DMF-DNN achieved 19.2%-25.2% lower RMSE than other machine learning algorithms.