A novel framework and concept-based semantic search Interface for abnormal crowd behaviour analysis in surveillance videos

Monitoring continuously captured surveillance videos is a challenging and time consuming task. To assist this issue, a new framework is introduced that applies anomaly detection, semantic annotation and provides a concept-based search interface. In particular, novel optical flow based features are used for abnormal crowd behaviour detection. Then, processed surveillance videos are annotated using a new semantic metadata model based on multimedia standards using Semantic Web technologies. In this way, globally inter-operable metadata about abnormal crowd behaviours are generated. Finally, for the first time, based on crowd behaviours, a novel concept-based semantic search interface is proposed. In the proposed interface, along with search results (video segments), statistical data about crowd behaviours are also presented. With extensive user studies, it is demonstrated that the proposed concept-based semantic search interface enables efficient search and analysis of abnormal crowd behaviours. Although there are existing works to achieve (a) crowd anomaly detection, (b) semantic annotation and (c) semantic search interface, none of the existing works combine these three system components in a novel framework like the one proposed in this paper. In each system component, we introduce contributions to the field as well as use the Semantic Web technologies to combine and standardize output of different system components; output of the anomaly detection is automatically annotated with metadata and stored to a semantic database. When continuous surveillance videos are processed, only the semantic database is updated. Finally, the user interface queries the updated database for searching/analyzing surveillance videos without changing any coding. Thus, the framework supports re-usability. This paper explains and evaluates different components of the framework.