Driver Emotion and Fatigue State Detection Based on Time Series Fusion

Studies have shown that driver fatigue or unpleasant emotions significantly increase driving risks. Detecting driver emotions and fatigue states and providing timely warnings can effectively minimize the incidence of traffic accidents. However, existing models rarely combine driver emotion and fatig...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Electronics (Basel) 2023-01, Vol.12 (1), p.26
Hauptverfasser: Shang, Yucheng, Yang, Mutian, Cui, Jianwei, Cui, Linwei, Huang, Zizheng, Li, Xiang
Format: Artikel
Sprache:eng
Schlagworte:
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page
container_issue 1
container_start_page 26
container_title Electronics (Basel)
container_volume 12
creator Shang, Yucheng
Yang, Mutian
Cui, Jianwei
Cui, Linwei
Huang, Zizheng
Li, Xiang
description Studies have shown that driver fatigue or unpleasant emotions significantly increase driving risks. Detecting driver emotions and fatigue states and providing timely warnings can effectively minimize the incidence of traffic accidents. However, existing models rarely combine driver emotion and fatigue detection, and there is space to improve the accuracy of recognition. In this paper, we propose a non-invasive and efficient detection method for driver fatigue and emotional state, which is the first time to combine them in the detection of driver state. Firstly, the captured video image sequences are preprocessed, and Dlib (image open source processing library) is used to locate face regions and mark key points; secondly, facial features are extracted, and fatigue indicators, such as driver eye closure time (PERCLOS) and yawn frequency are calculated using the dual-threshold method and fused by mathematical methods; thirdly, an improved lightweight RM-Xception convolutional neural network is introduced to identify the driver’s emotional state; finally, the two indicators are fused based on time series to obtain a comprehensive score for evaluating the driver’s state. The results show that the fatigue detection algorithm proposed in this paper has high accuracy, and the accuracy of the emotion recognition network reaches an accuracy rate of 73.32% on the Fer2013 dataset. The composite score calculated based on time series fusion can comprehensively and accurately reflect the driver state in different environments and make a contribution to future research in the field of assisted safe driving.
doi_str_mv 10.3390/electronics12010026
format Article
fullrecord <record><control><sourceid>gale_proqu</sourceid><recordid>TN_cdi_proquest_journals_2761112895</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><galeid>A745598457</galeid><sourcerecordid>A745598457</sourcerecordid><originalsourceid>FETCH-LOGICAL-c389t-2ac8234b1777db53c6a18a3cfbcca6161cc010ff46305bc77234232d13477b213</originalsourceid><addsrcrecordid>eNptUU1LAzEQDaJg0f4CLwuet-Zjd7M51rarQsGD9Ryys0lJ6W5qkhX896ZUUKEzh3nMezPD8BC6I3jGmMAPeq8hejdYCIRigjGtLtCEYi5yQQW9_IOv0TSEHU4hCKsZnqBm6e2n9tmqd9G6IVNDlzUq2u2os7eoos6WOqb9R-5RBd1lCWxsn1jtrQ5ZM4bE3aIro_ZBT3_qDXpvVpvFc75-fXpZzNc5sFrEnCqoKStawjnv2pJBpUitGJgWQFWkIgDpAWOKiuGyBc6TmDLaEVZw3lLCbtD9ae_Bu49Rhyh3bvRDOikprwghtBblr2qr9lrawbjoFfQ2gJzzoixFXZQ8qWZnVCk73VtwgzY29f8NsNMAeBeC10YevO2V_5IEy6MV8owV7BsIeHwE</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2761112895</pqid></control><display><type>article</type><title>Driver Emotion and Fatigue State Detection Based on Time Series Fusion</title><source>MDPI - Multidisciplinary Digital Publishing Institute</source><source>EZB-FREE-00999 freely available EZB journals</source><creator>Shang, Yucheng ; Yang, Mutian ; Cui, Jianwei ; Cui, Linwei ; Huang, Zizheng ; Li, Xiang</creator><creatorcontrib>Shang, Yucheng ; Yang, Mutian ; Cui, Jianwei ; Cui, Linwei ; Huang, Zizheng ; Li, Xiang</creatorcontrib><description>Studies have shown that driver fatigue or unpleasant emotions significantly increase driving risks. Detecting driver emotions and fatigue states and providing timely warnings can effectively minimize the incidence of traffic accidents. However, existing models rarely combine driver emotion and fatigue detection, and there is space to improve the accuracy of recognition. In this paper, we propose a non-invasive and efficient detection method for driver fatigue and emotional state, which is the first time to combine them in the detection of driver state. Firstly, the captured video image sequences are preprocessed, and Dlib (image open source processing library) is used to locate face regions and mark key points; secondly, facial features are extracted, and fatigue indicators, such as driver eye closure time (PERCLOS) and yawn frequency are calculated using the dual-threshold method and fused by mathematical methods; thirdly, an improved lightweight RM-Xception convolutional neural network is introduced to identify the driver’s emotional state; finally, the two indicators are fused based on time series to obtain a comprehensive score for evaluating the driver’s state. The results show that the fatigue detection algorithm proposed in this paper has high accuracy, and the accuracy of the emotion recognition network reaches an accuracy rate of 73.32% on the Fer2013 dataset. The composite score calculated based on time series fusion can comprehensively and accurately reflect the driver state in different environments and make a contribution to future research in the field of assisted safe driving.</description><identifier>ISSN: 2079-9292</identifier><identifier>EISSN: 2079-9292</identifier><identifier>DOI: 10.3390/electronics12010026</identifier><language>eng</language><publisher>Basel: MDPI AG</publisher><subject>Accuracy ; Algorithms ; Artificial neural networks ; Automobile drivers ; Classification ; Datasets ; Driver fatigue ; Electroencephalography ; Emotion recognition ; Emotional factors ; Emotions ; Fatigue ; Feature extraction ; Image processing ; Indicators ; Methods ; Neural networks ; Object recognition (Computers) ; Pattern recognition ; Physiology ; Psychological aspects ; Sequences ; Time series ; Traffic accidents ; Traffic models ; Traffic safety ; Wavelet transforms</subject><ispartof>Electronics (Basel), 2023-01, Vol.12 (1), p.26</ispartof><rights>COPYRIGHT 2022 MDPI AG</rights><rights>2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c389t-2ac8234b1777db53c6a18a3cfbcca6161cc010ff46305bc77234232d13477b213</citedby><cites>FETCH-LOGICAL-c389t-2ac8234b1777db53c6a18a3cfbcca6161cc010ff46305bc77234232d13477b213</cites><orcidid>0000-0003-3904-6324 ; 0000-0002-7335-1587</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>314,780,784,27924,27925</link.rule.ids></links><search><creatorcontrib>Shang, Yucheng</creatorcontrib><creatorcontrib>Yang, Mutian</creatorcontrib><creatorcontrib>Cui, Jianwei</creatorcontrib><creatorcontrib>Cui, Linwei</creatorcontrib><creatorcontrib>Huang, Zizheng</creatorcontrib><creatorcontrib>Li, Xiang</creatorcontrib><title>Driver Emotion and Fatigue State Detection Based on Time Series Fusion</title><title>Electronics (Basel)</title><description>Studies have shown that driver fatigue or unpleasant emotions significantly increase driving risks. Detecting driver emotions and fatigue states and providing timely warnings can effectively minimize the incidence of traffic accidents. However, existing models rarely combine driver emotion and fatigue detection, and there is space to improve the accuracy of recognition. In this paper, we propose a non-invasive and efficient detection method for driver fatigue and emotional state, which is the first time to combine them in the detection of driver state. Firstly, the captured video image sequences are preprocessed, and Dlib (image open source processing library) is used to locate face regions and mark key points; secondly, facial features are extracted, and fatigue indicators, such as driver eye closure time (PERCLOS) and yawn frequency are calculated using the dual-threshold method and fused by mathematical methods; thirdly, an improved lightweight RM-Xception convolutional neural network is introduced to identify the driver’s emotional state; finally, the two indicators are fused based on time series to obtain a comprehensive score for evaluating the driver’s state. The results show that the fatigue detection algorithm proposed in this paper has high accuracy, and the accuracy of the emotion recognition network reaches an accuracy rate of 73.32% on the Fer2013 dataset. The composite score calculated based on time series fusion can comprehensively and accurately reflect the driver state in different environments and make a contribution to future research in the field of assisted safe driving.</description><subject>Accuracy</subject><subject>Algorithms</subject><subject>Artificial neural networks</subject><subject>Automobile drivers</subject><subject>Classification</subject><subject>Datasets</subject><subject>Driver fatigue</subject><subject>Electroencephalography</subject><subject>Emotion recognition</subject><subject>Emotional factors</subject><subject>Emotions</subject><subject>Fatigue</subject><subject>Feature extraction</subject><subject>Image processing</subject><subject>Indicators</subject><subject>Methods</subject><subject>Neural networks</subject><subject>Object recognition (Computers)</subject><subject>Pattern recognition</subject><subject>Physiology</subject><subject>Psychological aspects</subject><subject>Sequences</subject><subject>Time series</subject><subject>Traffic accidents</subject><subject>Traffic models</subject><subject>Traffic safety</subject><subject>Wavelet transforms</subject><issn>2079-9292</issn><issn>2079-9292</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2023</creationdate><recordtype>article</recordtype><sourceid>ABUWG</sourceid><sourceid>AFKRA</sourceid><sourceid>AZQEC</sourceid><sourceid>BENPR</sourceid><sourceid>CCPQU</sourceid><sourceid>DWQXO</sourceid><recordid>eNptUU1LAzEQDaJg0f4CLwuet-Zjd7M51rarQsGD9Ryys0lJ6W5qkhX896ZUUKEzh3nMezPD8BC6I3jGmMAPeq8hejdYCIRigjGtLtCEYi5yQQW9_IOv0TSEHU4hCKsZnqBm6e2n9tmqd9G6IVNDlzUq2u2os7eoos6WOqb9R-5RBd1lCWxsn1jtrQ5ZM4bE3aIro_ZBT3_qDXpvVpvFc75-fXpZzNc5sFrEnCqoKStawjnv2pJBpUitGJgWQFWkIgDpAWOKiuGyBc6TmDLaEVZw3lLCbtD9ae_Bu49Rhyh3bvRDOikprwghtBblr2qr9lrawbjoFfQ2gJzzoixFXZQ8qWZnVCk73VtwgzY29f8NsNMAeBeC10YevO2V_5IEy6MV8owV7BsIeHwE</recordid><startdate>20230101</startdate><enddate>20230101</enddate><creator>Shang, Yucheng</creator><creator>Yang, Mutian</creator><creator>Cui, Jianwei</creator><creator>Cui, Linwei</creator><creator>Huang, Zizheng</creator><creator>Li, Xiang</creator><general>MDPI AG</general><scope>AAYXX</scope><scope>CITATION</scope><scope>7SP</scope><scope>8FD</scope><scope>8FE</scope><scope>8FG</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>ARAPS</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>HCIFZ</scope><scope>L7M</scope><scope>P5Z</scope><scope>P62</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><orcidid>https://orcid.org/0000-0003-3904-6324</orcidid><orcidid>https://orcid.org/0000-0002-7335-1587</orcidid></search><sort><creationdate>20230101</creationdate><title>Driver Emotion and Fatigue State Detection Based on Time Series Fusion</title><author>Shang, Yucheng ; Yang, Mutian ; Cui, Jianwei ; Cui, Linwei ; Huang, Zizheng ; Li, Xiang</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c389t-2ac8234b1777db53c6a18a3cfbcca6161cc010ff46305bc77234232d13477b213</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2023</creationdate><topic>Accuracy</topic><topic>Algorithms</topic><topic>Artificial neural networks</topic><topic>Automobile drivers</topic><topic>Classification</topic><topic>Datasets</topic><topic>Driver fatigue</topic><topic>Electroencephalography</topic><topic>Emotion recognition</topic><topic>Emotional factors</topic><topic>Emotions</topic><topic>Fatigue</topic><topic>Feature extraction</topic><topic>Image processing</topic><topic>Indicators</topic><topic>Methods</topic><topic>Neural networks</topic><topic>Object recognition (Computers)</topic><topic>Pattern recognition</topic><topic>Physiology</topic><topic>Psychological aspects</topic><topic>Sequences</topic><topic>Time series</topic><topic>Traffic accidents</topic><topic>Traffic models</topic><topic>Traffic safety</topic><topic>Wavelet transforms</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Shang, Yucheng</creatorcontrib><creatorcontrib>Yang, Mutian</creatorcontrib><creatorcontrib>Cui, Jianwei</creatorcontrib><creatorcontrib>Cui, Linwei</creatorcontrib><creatorcontrib>Huang, Zizheng</creatorcontrib><creatorcontrib>Li, Xiang</creatorcontrib><collection>CrossRef</collection><collection>Electronics &amp; Communications Abstracts</collection><collection>Technology Research Database</collection><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>ProQuest Central (Alumni Edition)</collection><collection>ProQuest Central UK/Ireland</collection><collection>Advanced Technologies &amp; Aerospace Collection</collection><collection>ProQuest Central Essentials</collection><collection>ProQuest Central</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central Korea</collection><collection>SciTech Premium Collection</collection><collection>Advanced Technologies Database with Aerospace</collection><collection>Advanced Technologies &amp; Aerospace Database</collection><collection>ProQuest Advanced Technologies &amp; Aerospace Collection</collection><collection>Publicly Available Content Database</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><jtitle>Electronics (Basel)</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Shang, Yucheng</au><au>Yang, Mutian</au><au>Cui, Jianwei</au><au>Cui, Linwei</au><au>Huang, Zizheng</au><au>Li, Xiang</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Driver Emotion and Fatigue State Detection Based on Time Series Fusion</atitle><jtitle>Electronics (Basel)</jtitle><date>2023-01-01</date><risdate>2023</risdate><volume>12</volume><issue>1</issue><spage>26</spage><pages>26-</pages><issn>2079-9292</issn><eissn>2079-9292</eissn><abstract>Studies have shown that driver fatigue or unpleasant emotions significantly increase driving risks. Detecting driver emotions and fatigue states and providing timely warnings can effectively minimize the incidence of traffic accidents. However, existing models rarely combine driver emotion and fatigue detection, and there is space to improve the accuracy of recognition. In this paper, we propose a non-invasive and efficient detection method for driver fatigue and emotional state, which is the first time to combine them in the detection of driver state. Firstly, the captured video image sequences are preprocessed, and Dlib (image open source processing library) is used to locate face regions and mark key points; secondly, facial features are extracted, and fatigue indicators, such as driver eye closure time (PERCLOS) and yawn frequency are calculated using the dual-threshold method and fused by mathematical methods; thirdly, an improved lightweight RM-Xception convolutional neural network is introduced to identify the driver’s emotional state; finally, the two indicators are fused based on time series to obtain a comprehensive score for evaluating the driver’s state. The results show that the fatigue detection algorithm proposed in this paper has high accuracy, and the accuracy of the emotion recognition network reaches an accuracy rate of 73.32% on the Fer2013 dataset. The composite score calculated based on time series fusion can comprehensively and accurately reflect the driver state in different environments and make a contribution to future research in the field of assisted safe driving.</abstract><cop>Basel</cop><pub>MDPI AG</pub><doi>10.3390/electronics12010026</doi><orcidid>https://orcid.org/0000-0003-3904-6324</orcidid><orcidid>https://orcid.org/0000-0002-7335-1587</orcidid><oa>free_for_read</oa></addata></record>
fulltext fulltext
identifier ISSN: 2079-9292
ispartof Electronics (Basel), 2023-01, Vol.12 (1), p.26
issn 2079-9292
2079-9292
language eng
recordid cdi_proquest_journals_2761112895
source MDPI - Multidisciplinary Digital Publishing Institute; EZB-FREE-00999 freely available EZB journals
subjects Accuracy
Algorithms
Artificial neural networks
Automobile drivers
Classification
Datasets
Driver fatigue
Electroencephalography
Emotion recognition
Emotional factors
Emotions
Fatigue
Feature extraction
Image processing
Indicators
Methods
Neural networks
Object recognition (Computers)
Pattern recognition
Physiology
Psychological aspects
Sequences
Time series
Traffic accidents
Traffic models
Traffic safety
Wavelet transforms
title Driver Emotion and Fatigue State Detection Based on Time Series Fusion
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-07T09%3A53%3A37IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-gale_proqu&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Driver%20Emotion%20and%20Fatigue%20State%20Detection%20Based%20on%20Time%20Series%20Fusion&rft.jtitle=Electronics%20(Basel)&rft.au=Shang,%20Yucheng&rft.date=2023-01-01&rft.volume=12&rft.issue=1&rft.spage=26&rft.pages=26-&rft.issn=2079-9292&rft.eissn=2079-9292&rft_id=info:doi/10.3390/electronics12010026&rft_dat=%3Cgale_proqu%3EA745598457%3C/gale_proqu%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2761112895&rft_id=info:pmid/&rft_galeid=A745598457&rfr_iscdi=true