Deep multi-person kinship matching and recognition for family photos

•First, we design a deep kinship matching and recognition (DKMR) framework for understanding kinship in a nuclear family automatically. It makes the first attempt to generate a nuclear family tree end-to-end, to our best knowledge.•Second, compared with previous kinship understanding methods focusing on pairwise face images, 70 the input of our DKMR framework extends to one nuclear family photo. The experiments on Group-Face, TSkinFace and FIW datasets demonstrate its effectiveness.•Third, our proposed reasoning conditional random field (R-CRF) algorithm fully exploits the common kinship rules to well boost matching and recognition accuracy and ensure the output family tree is optimal. In this paper, we propose a novel Deep Kinship Matching and Recognition (DKMR) framework for multi-person kinship matching and recognition, which is a complicated and challenging task with little previous literature. Compared with most existing kinship understanding methods that mainly work on matching kinship in pairwise face images, we target at recognizing the exact kinship in nuclear family photos consisting of multiple persons. The proposed DKMR framework contains three modules. Firstly, we design a deep kinship matching model (termed DKM-TRL) to predict kin-or-not scores by integrating the triple ranking loss into a Siamese CNN model. Secondly, we develop a deep kinship recognition model (named DKR-GA) to predict the exact kinship categories, in which gender and relative age attributes are utilized to learn more discriminative representations. Thirdly, based on the outputs of DKM-TRL and DKR-GA, we propose a reasoning conditional random field (R-CRF) model to infer the corresponding optimal family tree by exploiting the common kinship knowledge of a nuclear family. To evaluate the effectiveness of our DKMR framework, we conduct extensive experiments and the results show that it can gain superior performance on Group-Face dataset, TSKinFace dataset and FIW dataset over state-of-the-arts.