Quality control of online monitoring data of air pollutants using artificial neural networks
The intensive monitoring of air pollutants has led to the acquisition of vast quantities of data. Traditional quality control methods based on existing knowledge may be inefficient because of our limited understanding regarding the interaction of human activities and stochastic environmental factors...
Gespeichert in:
Veröffentlicht in: | Air quality, atmosphere and health atmosphere and health, 2019-10, Vol.12 (10), p.1189-1196 |
---|---|
Hauptverfasser: | , , , , , |
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
container_end_page | 1196 |
---|---|
container_issue | 10 |
container_start_page | 1189 |
container_title | Air quality, atmosphere and health |
container_volume | 12 |
creator | Wang, Ziyu Feng, Jingjing Fu, Qingyan Gao, Song Chen, Xiaojia Cheng, Jinping |
description | The intensive monitoring of air pollutants has led to the acquisition of vast quantities of data. Traditional quality control methods based on existing knowledge may be inefficient because of our limited understanding regarding the interaction of human activities and stochastic environmental factors. Moreover, traditional methods for outlier detection may be misleading because of the existence of valid outliers and invalid inliers. In this research, artificial neural networks (ANNs) are developed to identify instrument failure based on current and historical observations. Two structures, i.e., multilayer perceptrons and recurrent networks, are trained using 50,000 hourly data points labeled by human reviewers. The most conservative model identified 57.5% of the invalid sulfur compound observations and 44.9% of the invalid nitrogen compound observations. By setting a more liberal threshold, these values increased to 76.0% and 79.7%, respectively. Except for SO
2
, the ANNs outperformed the traditional methods for data quality control, as demonstrated with a plausibility test, a test of temporal consistency and a residential analysis. Compared with the test of temporal consistency, which was the most effective traditional method studied, the true positive rates of the ANNs were 19.4% to 29.5% higher for all pollutants except SO
2
, given the same false positive rates. The results indicate the effectiveness of ANNs for data quality control even without supplementary information. Methods for performance improvement are discussed. |
doi_str_mv | 10.1007/s11869-019-00734-4 |
format | Article |
fullrecord | <record><control><sourceid>proquest_cross</sourceid><recordid>TN_cdi_proquest_journals_2269228499</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2269228499</sourcerecordid><originalsourceid>FETCH-LOGICAL-c367t-d3a85dad0a335ed564b1adbb560b85517c5f39b4a50130662867325c5ad94a2b3</originalsourceid><addsrcrecordid>eNp9kE9LxDAQxYMouK5-AU8Bz9X8b3uURV1hQQS9CWHappK1m9QkRfbb225Fbx6GNzDvvYEfQpeUXFNC8ptIaaHKjNBxSM5FJo7QghY5z0rO1PHvTotTdBbjlhBFBFEL9PY8QGfTHtfepeA77FvsXWedwTvvbPLBunfcQILpAjbg3nfdkMCliIc4HSEk29raQoedGcJB0pcPH_EcnbTQRXPxo0v0en_3slpnm6eHx9XtJqu5ylPWcChkAw0BzqVppBIVhaaqpCJVISXNa9nyshIgCeVEKVaonDNZS2hKAaziS3Q19_bBfw4mJr31Q3DjS82YKhkrRFmOLja76uBjDKbVfbA7CHtNiZ4o6pmiHinqA0UtxhCfQ7GfSJjwV_1P6huFqnYx</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2269228499</pqid></control><display><type>article</type><title>Quality control of online monitoring data of air pollutants using artificial neural networks</title><source>SpringerLink Journals - AutoHoldings</source><creator>Wang, Ziyu ; Feng, Jingjing ; Fu, Qingyan ; Gao, Song ; Chen, Xiaojia ; Cheng, Jinping</creator><creatorcontrib>Wang, Ziyu ; Feng, Jingjing ; Fu, Qingyan ; Gao, Song ; Chen, Xiaojia ; Cheng, Jinping</creatorcontrib><description>The intensive monitoring of air pollutants has led to the acquisition of vast quantities of data. Traditional quality control methods based on existing knowledge may be inefficient because of our limited understanding regarding the interaction of human activities and stochastic environmental factors. Moreover, traditional methods for outlier detection may be misleading because of the existence of valid outliers and invalid inliers. In this research, artificial neural networks (ANNs) are developed to identify instrument failure based on current and historical observations. Two structures, i.e., multilayer perceptrons and recurrent networks, are trained using 50,000 hourly data points labeled by human reviewers. The most conservative model identified 57.5% of the invalid sulfur compound observations and 44.9% of the invalid nitrogen compound observations. By setting a more liberal threshold, these values increased to 76.0% and 79.7%, respectively. Except for SO
2
, the ANNs outperformed the traditional methods for data quality control, as demonstrated with a plausibility test, a test of temporal consistency and a residential analysis. Compared with the test of temporal consistency, which was the most effective traditional method studied, the true positive rates of the ANNs were 19.4% to 29.5% higher for all pollutants except SO
2
, given the same false positive rates. The results indicate the effectiveness of ANNs for data quality control even without supplementary information. Methods for performance improvement are discussed.</description><identifier>ISSN: 1873-9318</identifier><identifier>EISSN: 1873-9326</identifier><identifier>DOI: 10.1007/s11869-019-00734-4</identifier><language>eng</language><publisher>Dordrecht: Springer Netherlands</publisher><subject>Air monitoring ; Air pollution ; Artificial neural networks ; Atmospheric Protection/Air Quality Control/Air Pollution ; Consistency ; Control methods ; Data analysis ; Data points ; Data quality control ; Earth and Environmental Science ; Environment ; Environmental factors ; Environmental Health ; Environmental monitoring ; Health Promotion and Disease Prevention ; Historical structures ; Inliers (landforms) ; Multilayer perceptrons ; Neural networks ; Nitrogen compounds ; Outliers (statistics) ; Pollutants ; Pollution monitoring ; Quality control ; Stochasticity ; Sulfur ; Sulfur compounds ; Sulfur dioxide</subject><ispartof>Air quality, atmosphere and health, 2019-10, Vol.12 (10), p.1189-1196</ispartof><rights>Springer Nature B.V. 2019</rights><rights>Air Quality, Atmosphere and Health is a copyright of Springer, (2019). All Rights Reserved.</rights><lds50>peer_reviewed</lds50><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c367t-d3a85dad0a335ed564b1adbb560b85517c5f39b4a50130662867325c5ad94a2b3</citedby><cites>FETCH-LOGICAL-c367t-d3a85dad0a335ed564b1adbb560b85517c5f39b4a50130662867325c5ad94a2b3</cites></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktopdf>$$Uhttps://link.springer.com/content/pdf/10.1007/s11869-019-00734-4$$EPDF$$P50$$Gspringer$$H</linktopdf><linktohtml>$$Uhttps://link.springer.com/10.1007/s11869-019-00734-4$$EHTML$$P50$$Gspringer$$H</linktohtml><link.rule.ids>314,780,784,27924,27925,41488,42557,51319</link.rule.ids></links><search><creatorcontrib>Wang, Ziyu</creatorcontrib><creatorcontrib>Feng, Jingjing</creatorcontrib><creatorcontrib>Fu, Qingyan</creatorcontrib><creatorcontrib>Gao, Song</creatorcontrib><creatorcontrib>Chen, Xiaojia</creatorcontrib><creatorcontrib>Cheng, Jinping</creatorcontrib><title>Quality control of online monitoring data of air pollutants using artificial neural networks</title><title>Air quality, atmosphere and health</title><addtitle>Air Qual Atmos Health</addtitle><description>The intensive monitoring of air pollutants has led to the acquisition of vast quantities of data. Traditional quality control methods based on existing knowledge may be inefficient because of our limited understanding regarding the interaction of human activities and stochastic environmental factors. Moreover, traditional methods for outlier detection may be misleading because of the existence of valid outliers and invalid inliers. In this research, artificial neural networks (ANNs) are developed to identify instrument failure based on current and historical observations. Two structures, i.e., multilayer perceptrons and recurrent networks, are trained using 50,000 hourly data points labeled by human reviewers. The most conservative model identified 57.5% of the invalid sulfur compound observations and 44.9% of the invalid nitrogen compound observations. By setting a more liberal threshold, these values increased to 76.0% and 79.7%, respectively. Except for SO
2
, the ANNs outperformed the traditional methods for data quality control, as demonstrated with a plausibility test, a test of temporal consistency and a residential analysis. Compared with the test of temporal consistency, which was the most effective traditional method studied, the true positive rates of the ANNs were 19.4% to 29.5% higher for all pollutants except SO
2
, given the same false positive rates. The results indicate the effectiveness of ANNs for data quality control even without supplementary information. Methods for performance improvement are discussed.</description><subject>Air monitoring</subject><subject>Air pollution</subject><subject>Artificial neural networks</subject><subject>Atmospheric Protection/Air Quality Control/Air Pollution</subject><subject>Consistency</subject><subject>Control methods</subject><subject>Data analysis</subject><subject>Data points</subject><subject>Data quality control</subject><subject>Earth and Environmental Science</subject><subject>Environment</subject><subject>Environmental factors</subject><subject>Environmental Health</subject><subject>Environmental monitoring</subject><subject>Health Promotion and Disease Prevention</subject><subject>Historical structures</subject><subject>Inliers (landforms)</subject><subject>Multilayer perceptrons</subject><subject>Neural networks</subject><subject>Nitrogen compounds</subject><subject>Outliers (statistics)</subject><subject>Pollutants</subject><subject>Pollution monitoring</subject><subject>Quality control</subject><subject>Stochasticity</subject><subject>Sulfur</subject><subject>Sulfur compounds</subject><subject>Sulfur dioxide</subject><issn>1873-9318</issn><issn>1873-9326</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2019</creationdate><recordtype>article</recordtype><sourceid>ABUWG</sourceid><sourceid>AFKRA</sourceid><sourceid>AZQEC</sourceid><sourceid>BENPR</sourceid><sourceid>CCPQU</sourceid><sourceid>DWQXO</sourceid><sourceid>GNUQQ</sourceid><recordid>eNp9kE9LxDAQxYMouK5-AU8Bz9X8b3uURV1hQQS9CWHappK1m9QkRfbb225Fbx6GNzDvvYEfQpeUXFNC8ptIaaHKjNBxSM5FJo7QghY5z0rO1PHvTotTdBbjlhBFBFEL9PY8QGfTHtfepeA77FvsXWedwTvvbPLBunfcQILpAjbg3nfdkMCliIc4HSEk29raQoedGcJB0pcPH_EcnbTQRXPxo0v0en_3slpnm6eHx9XtJqu5ylPWcChkAw0BzqVppBIVhaaqpCJVISXNa9nyshIgCeVEKVaonDNZS2hKAaziS3Q19_bBfw4mJr31Q3DjS82YKhkrRFmOLja76uBjDKbVfbA7CHtNiZ4o6pmiHinqA0UtxhCfQ7GfSJjwV_1P6huFqnYx</recordid><startdate>20191001</startdate><enddate>20191001</enddate><creator>Wang, Ziyu</creator><creator>Feng, Jingjing</creator><creator>Fu, Qingyan</creator><creator>Gao, Song</creator><creator>Chen, Xiaojia</creator><creator>Cheng, Jinping</creator><general>Springer Netherlands</general><general>Springer Nature B.V</general><scope>AAYXX</scope><scope>CITATION</scope><scope>3V.</scope><scope>7T2</scope><scope>7TG</scope><scope>7TV</scope><scope>7WY</scope><scope>7WZ</scope><scope>7X7</scope><scope>7XB</scope><scope>87Z</scope><scope>88I</scope><scope>8C1</scope><scope>8FI</scope><scope>8FJ</scope><scope>8FK</scope><scope>8FL</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>ATCPS</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BEZIV</scope><scope>BHPHI</scope><scope>BKSAR</scope><scope>C1K</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>FRNLG</scope><scope>FYUFA</scope><scope>F~G</scope><scope>GHDGH</scope><scope>GNUQQ</scope><scope>HCIFZ</scope><scope>K60</scope><scope>K6~</scope><scope>K9.</scope><scope>KL.</scope><scope>L.-</scope><scope>M0C</scope><scope>M0S</scope><scope>M2P</scope><scope>PATMY</scope><scope>PCBAR</scope><scope>PQBIZ</scope><scope>PQBZA</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PYCSY</scope><scope>Q9U</scope></search><sort><creationdate>20191001</creationdate><title>Quality control of online monitoring data of air pollutants using artificial neural networks</title><author>Wang, Ziyu ; Feng, Jingjing ; Fu, Qingyan ; Gao, Song ; Chen, Xiaojia ; Cheng, Jinping</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c367t-d3a85dad0a335ed564b1adbb560b85517c5f39b4a50130662867325c5ad94a2b3</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2019</creationdate><topic>Air monitoring</topic><topic>Air pollution</topic><topic>Artificial neural networks</topic><topic>Atmospheric Protection/Air Quality Control/Air Pollution</topic><topic>Consistency</topic><topic>Control methods</topic><topic>Data analysis</topic><topic>Data points</topic><topic>Data quality control</topic><topic>Earth and Environmental Science</topic><topic>Environment</topic><topic>Environmental factors</topic><topic>Environmental Health</topic><topic>Environmental monitoring</topic><topic>Health Promotion and Disease Prevention</topic><topic>Historical structures</topic><topic>Inliers (landforms)</topic><topic>Multilayer perceptrons</topic><topic>Neural networks</topic><topic>Nitrogen compounds</topic><topic>Outliers (statistics)</topic><topic>Pollutants</topic><topic>Pollution monitoring</topic><topic>Quality control</topic><topic>Stochasticity</topic><topic>Sulfur</topic><topic>Sulfur compounds</topic><topic>Sulfur dioxide</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Wang, Ziyu</creatorcontrib><creatorcontrib>Feng, Jingjing</creatorcontrib><creatorcontrib>Fu, Qingyan</creatorcontrib><creatorcontrib>Gao, Song</creatorcontrib><creatorcontrib>Chen, Xiaojia</creatorcontrib><creatorcontrib>Cheng, Jinping</creatorcontrib><collection>CrossRef</collection><collection>ProQuest Central (Corporate)</collection><collection>Health and Safety Science Abstracts (Full archive)</collection><collection>Meteorological & Geoastrophysical Abstracts</collection><collection>Pollution Abstracts</collection><collection>ABI/INFORM Collection</collection><collection>ABI/INFORM Global (PDF only)</collection><collection>Health & Medical Collection</collection><collection>ProQuest Central (purchase pre-March 2016)</collection><collection>ABI/INFORM Global (Alumni Edition)</collection><collection>Science Database (Alumni Edition)</collection><collection>Public Health Database</collection><collection>Hospital Premium Collection</collection><collection>Hospital Premium Collection (Alumni Edition)</collection><collection>ProQuest Central (Alumni) (purchase pre-March 2016)</collection><collection>ABI/INFORM Collection (Alumni Edition)</collection><collection>ProQuest Central (Alumni Edition)</collection><collection>ProQuest Central UK/Ireland</collection><collection>Agricultural & Environmental Science Collection</collection><collection>ProQuest Central Essentials</collection><collection>ProQuest Central</collection><collection>Business Premium Collection</collection><collection>Natural Science Collection</collection><collection>Earth, Atmospheric & Aquatic Science Collection</collection><collection>Environmental Sciences and Pollution Management</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central Korea</collection><collection>Business Premium Collection (Alumni)</collection><collection>Health Research Premium Collection</collection><collection>ABI/INFORM Global (Corporate)</collection><collection>Health Research Premium Collection (Alumni)</collection><collection>ProQuest Central Student</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Business Collection (Alumni Edition)</collection><collection>ProQuest Business Collection</collection><collection>ProQuest Health & Medical Complete (Alumni)</collection><collection>Meteorological & Geoastrophysical Abstracts - Academic</collection><collection>ABI/INFORM Professional Advanced</collection><collection>ABI/INFORM Global</collection><collection>Health & Medical Collection (Alumni Edition)</collection><collection>Science Database</collection><collection>Environmental Science Database</collection><collection>Earth, Atmospheric & Aquatic Science Database</collection><collection>ProQuest One Business</collection><collection>ProQuest One Business (Alumni)</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>Environmental Science Collection</collection><collection>ProQuest Central Basic</collection><jtitle>Air quality, atmosphere and health</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Wang, Ziyu</au><au>Feng, Jingjing</au><au>Fu, Qingyan</au><au>Gao, Song</au><au>Chen, Xiaojia</au><au>Cheng, Jinping</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Quality control of online monitoring data of air pollutants using artificial neural networks</atitle><jtitle>Air quality, atmosphere and health</jtitle><stitle>Air Qual Atmos Health</stitle><date>2019-10-01</date><risdate>2019</risdate><volume>12</volume><issue>10</issue><spage>1189</spage><epage>1196</epage><pages>1189-1196</pages><issn>1873-9318</issn><eissn>1873-9326</eissn><abstract>The intensive monitoring of air pollutants has led to the acquisition of vast quantities of data. Traditional quality control methods based on existing knowledge may be inefficient because of our limited understanding regarding the interaction of human activities and stochastic environmental factors. Moreover, traditional methods for outlier detection may be misleading because of the existence of valid outliers and invalid inliers. In this research, artificial neural networks (ANNs) are developed to identify instrument failure based on current and historical observations. Two structures, i.e., multilayer perceptrons and recurrent networks, are trained using 50,000 hourly data points labeled by human reviewers. The most conservative model identified 57.5% of the invalid sulfur compound observations and 44.9% of the invalid nitrogen compound observations. By setting a more liberal threshold, these values increased to 76.0% and 79.7%, respectively. Except for SO
2
, the ANNs outperformed the traditional methods for data quality control, as demonstrated with a plausibility test, a test of temporal consistency and a residential analysis. Compared with the test of temporal consistency, which was the most effective traditional method studied, the true positive rates of the ANNs were 19.4% to 29.5% higher for all pollutants except SO
2
, given the same false positive rates. The results indicate the effectiveness of ANNs for data quality control even without supplementary information. Methods for performance improvement are discussed.</abstract><cop>Dordrecht</cop><pub>Springer Netherlands</pub><doi>10.1007/s11869-019-00734-4</doi><tpages>8</tpages></addata></record> |
fulltext | fulltext |
identifier | ISSN: 1873-9318 |
ispartof | Air quality, atmosphere and health, 2019-10, Vol.12 (10), p.1189-1196 |
issn | 1873-9318 1873-9326 |
language | eng |
recordid | cdi_proquest_journals_2269228499 |
source | SpringerLink Journals - AutoHoldings |
subjects | Air monitoring Air pollution Artificial neural networks Atmospheric Protection/Air Quality Control/Air Pollution Consistency Control methods Data analysis Data points Data quality control Earth and Environmental Science Environment Environmental factors Environmental Health Environmental monitoring Health Promotion and Disease Prevention Historical structures Inliers (landforms) Multilayer perceptrons Neural networks Nitrogen compounds Outliers (statistics) Pollutants Pollution monitoring Quality control Stochasticity Sulfur Sulfur compounds Sulfur dioxide |
title | Quality control of online monitoring data of air pollutants using artificial neural networks |
url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-24T18%3A11%3A42IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_cross&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Quality%20control%20of%20online%20monitoring%20data%20of%20air%20pollutants%20using%20artificial%20neural%20networks&rft.jtitle=Air%20quality,%20atmosphere%20and%20health&rft.au=Wang,%20Ziyu&rft.date=2019-10-01&rft.volume=12&rft.issue=10&rft.spage=1189&rft.epage=1196&rft.pages=1189-1196&rft.issn=1873-9318&rft.eissn=1873-9326&rft_id=info:doi/10.1007/s11869-019-00734-4&rft_dat=%3Cproquest_cross%3E2269228499%3C/proquest_cross%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2269228499&rft_id=info:pmid/&rfr_iscdi=true |