Weighted voting of multi-stream convolutional neural networks for video-based action recognition using optical flow rhythms

•We propose a multi-stream approach to solve the human action recognition problem.•The approach is based on the weighted voting of convolutional neural networks.•The voting weights are defined using a simulated annealing metaheuristic.•Multiple visual rhythms are employed to improve the spatio-temporal stream.•A novel temporal video representation is proposed to model long-term motion dynamics. Two of the most important premises of an ensemble are the diversity of its components and how to combine their votes. In this paper, we propose a multi-stream architecture based on the weighted voting of convolutional neural networks to deal with the problem of recognizing human actions in videos. A major challenge is how to include temporal aspects into this kind of approach. A key step in this direction is the selection of features that characterize the complexity of human actions in time. In this context, we propose a new stream, Optical Flow Rhythm, besides using other streams for diversity. To combine the streams, a voting system based on a new weighted average fusion method is introduced. In this scheme, the weights of classifiers are defined by an optimization process led by a metaheuristic. Experiments conducted on the UCF101 and HMDB51 datasets demonstrate that our method is comparable to state-of-the-art approaches.