A unified framework for local visual descriptors evaluation

Local descriptors are the ground layer of recognition feature based systems for still images and video. We propose a new framework for the design of local descriptors and their evaluation. This framework is based on the descriptors decomposition in three levels: primitive extraction, primitive coding and code aggregation. With this framework, we are able to explain most of the popular descriptors in the literature such as HOG, HOF or SURF. This framework provides an efficient and rigorous approach for the evaluation of local descriptors, and allows us to uncover the best parameters for each descriptor family. Moreover, we are able to extend usual descriptors by changing the code aggregation or adding new primitive coding method. The experiments are carried out on images (VOC 2007) and videos datasets (KTH, Hollywood2, UCF11 and UCF101), and achieve equal or better performances than the literature. •We propose a new framework to design local visual descriptors.•Descriptors are decomposed in primitive extraction, coding and aggregation.•Our framework can explain the most popular descriptors such as HOG, HOF, SURF.•The framework allows us a rigorous exploration of the possible combinations of primitives, coding and aggregation.•New descriptors are easily obtained by changing one of the steps of our framework.