iSecureHome: A deep fusion framework for surveillance of smart homes using real-time emotion recognition

With the advent of AI, the internet of things (IoT) and human-centric computing (HCC), the world has witnessed a rapid proliferation of smart homes (SH). However, implementing a robust security system for residents of SH remains a daunting task. The existing smart homes incorporate security provisions such as biometric verification, activity tracking, and facial recognition. Integrating multi-sensor devices, networking systems and data storage facilities escalate the lifecycle costs of these systems. Facial emotions convey important cues on behaviour and intent that can be used as non-invasive feedback for contextual threat analysis. The early mitigation of a hostile situation, such as a fight or an attempted intrusion, is vital for the SH residents’ safety. This research proposes a real-time facial emotion-based security framework called iSecureHome for smart homes using a CMOS camera, which is triggered by a passive infrared (PIR) motion sensor. The impact of chromatic and achromatic features on facial Emotion Recognition (ER), as well as skin colour-based biases in current ER algorithms, are also investigated. A time-bound facial emotion decoding strategy is presented in iSecureHome that is based on EmoFusioNet—a deep fusion-based model—to predict the security concerns in the vicinity of a given residence. EmoFusioNet utilises stacked and late fusion methodologies to ensure a colour-neutral and equitable ER system. Initially, the stacked model synchronously extracts the chromatic and achromatic facial features using deep CNNs, and their predictions are then fed into the late fusion component. After that, a regularised multi-layer perceptron (R-MLP) is trained to fuse the results of stacked CNNs and generate final predictions. Experimental results suggest that the proposed fusion methodology augments the ER model and achieves the final train and test accuracy of 98.48% and 98.43%, respectively. iSecureHome also comprises a multi-threaded decision-making framework for threat analysis with efficient performance and minimal latency. [Display omitted] •iSecureHome is an emotion-recognition (ER) based Smart Home security framework.•This framework employs a hybrid fusion model, EmoFusioNet, for colour-neutral ER.•EmoFusioNet aggregates chromatic and achromatic features for superior performance.•Performance evaluation of the EmoFusioNet model using public and collected datasets.•Real-time validation requires a decision-making module with a notification system.