Real‐Time Non‐Driving Behavior Recognition Using Deep Learning‐Assisted Triboelectric Sensors in Conditionally Automated Driving

Real‐time recognition of non‐driving behaviors is of great importance in conditionally automated driving, as it determines the takeover time budget, which in turn has a huge impact on the performance of the takeover. Here, a novel real‐time non‐driving behavior recognition system (RNBRS) integrating self‐powered, low‐cost, easy‐to‐manufacture triboelectric sensors, and a deep learning model is proposed. The structure, working mechanism, and electrical characteristics of triboelectric sensors are investigated and analyzed. Through the ingenious structural design of single‐electrode triboelectric sensors and driving simulation experiments under conditional automated driving, non‐driving behaviors are captured in the form of electrical signals. A well‐trained long short‐term memory network model is adopted to recognize the five most typical non‐driving behaviors, including phone, console touchpad, driving, monitoring driving, and no operation, and test accuracy of 93.5% is achieved. Demonstration of a set of controlled experiments shows that RNBRS enables vehicles with conditional automation to dynamically adjust takeover time budget based on driver behavior, therefore significantly improving both safety and stability of takeover. This study opens new frontiers for the development of self‐powered electronics and inspires new thoughts on human‐machine interaction and the safety of autonomous vehicles. Triboelectric sensors with ingenious structural designs are capable of capturing detailed movements from the hands of drivers. Combining a deep learning‐based multi‐class classifier and triboelectric sensors, real‐time recognition of typical non‐driving behaviors in conditionally automated driving is realized. According to the recognition results, the takeover time budget can be dynamically adjusted, therefore, improving the takeover performance in both safety and stability.