Design considerations for the processing system of a CNN-based automated surveillance system

•New algorithms have changed the implementation of automated surveillance systems.•Batches of camera frames better leverage modern processing hardware.•Quantization can be effective in reducing processing time.•Deep learning framework may impact performance. Modern surveillance systems combine tens or even hundreds of cameras to ensure a wide area of coverage. With such a large amount of incoming video data, it is difficult for humans to effectively monitor each camera for potential threats. Recent advances in machine learning have brought forth great improvements in computer vision, which forms the key to automating surveillance systems. Previous works on the topic of automated surveillance systems have not been updated to leverage the emerging wave of convolutional neural network (CNN) based algorithms, nor considered the implications of these rapidly evolving frameworks. This work investigates the design of the processing system for a CNN-based automated surveillance system, including off-the-shelf algorithms and hardware components. Performance benchmarks are then used to evaluate several design options that impact processing speed of CNN-based surveillance algorithms. Finally, a set of design considerations are presented for the processing system of a modern CNN-based automated surveillance system.