Object categorization with sketch representation and generalized samples

In this paper, we present a framework for object categorization via sketch graphs that incorporate shape and structure information. In this framework, we integrate the learnable And–Or graph model, a hierarchical structure that combines the reconfigurability of a stochastic context free grammar (SCFG) with the constraints of a Markov random field (MRF). Considering the computation efficiency, we generalize instances from the And–Or graph models and perform a set of sequential tests for cascaded object categorization, rather than directly inferring with the And–Or graph models. We study 33 categories, each consisting of a small data set of 30 instances, and 30 additional templates with varied appearance are generalized from the learned And–Or graph model. These samples better span the appearance space and form an augmented training set ΩT of 1980 (60×33) training templates. To perform recognition on a testing image, we use a set of sequential tests to project ΩT into different representation spaces to narrow the number of candidate matches in ΩT. We use “graphlets” (structural elements), as our local features and model ΩT at each stage using histograms of graphlets over categories, histograms of graphlets over object instances, histograms of pairs of graphlets over objects, and shape context. Each test is increasingly computationally expensive, and by the end of the cascade we have a small candidate set remaining to use with our most powerful test, a top-down graph matching algorithm. We apply the proposed approach on the challenging public dataset including 33 object categories, and achieve state-of-the-art performance. ► We present a framework for object categorization via sketch graphs. ► We generate samples from the learnable And–Or graph models for training. ► We perform a set of sequential tests for cascaded object categorization. ► Our system achieves 81.4% classification rate in 33 object categories.