CNN based spatial classification features for clustering offline handwritten mathematical expressions

•High performance of clustering images with 0.99 purity on CROHME.•Hierarchical representation of spatial classification features for clustering.•Propose a novel global pooling method for weakly supervised learning.•Classifying and localizing multi-scale symbols of handwritten images. To help human markers mark a large number of answers of handwritten mathematical expressions (HMEs), clustering them makes marking more efficient and reliable. Clustering HMEs, however, faces the problem of extracting both localization and classification representation of mathematical symbols for an HME image and defining the distance between two HME images. First, we propose a method based on Convolutional Neural Networks (CNN) to extract the representations for an HME. Symbols in various scales are located and classified by a combination of features from a multi-scale CNN. We use weakly supervised training combined with symbols attention to enhance localization and classification predictions. Second, we propose a multi-level spatial distance between two representations for clustering HMEs. Experiments on CROHME 2016 and CROHME 2019 dataset show the promising results of 0.99 and 0.96 in purity, respectively.