Changepoint detection-assisted nonparametric clustering for unsupervised temporal sign segmentation

Temporal sign segmentation aims to temporarily divide continuous sign language into category-agnostic sign segments. One of the challenges to temporal sign segmentation is the paucity of frame-level annotations for sign language videos, which restricts the applicability of supervised and semi-supervised approaches. To address this challenge, we consider temporal sign segmentation as a clustering problem defined as the grouping of semantically similar frames of a given video, which allows us to choose unsupervised clustering techniques as promising alternatives. However, most unsupervised clustering techniques are parametric. In the context of temporal sign segmentation, they assume that the number of sign segments of a given continuous sign video is predefined, which is implausible. The primary contributions of this study are as follows: (1) We propose the first nonparametric clustering algorithm for unsupervised temporal sign segmentation. The main concept is to enhance the hierarchical graph-based clustering algorithm to be nonparametric by adopting cost and penalty functions of a changepoint detection algorithm to determine the optimal number of sign segments. Experimental results show that the performance of the proposed unsupervised method is comparable to that of the latest semi-supervised sign segmentation method in terms of several metrics. Moreover, the execution time of the proposed clustering method was less than 1 s, thereby ensuring its applicability. (2) We identify that the conventional metrics for temporal sign segmentation do not sufficiently address over-segmentation. To overcome the difficulty, we propose a new main metric to evaluate the performance of temporal sign segmentation, called the adjusted MF1B. •We propose the CDNC algorithm for unsupervised temporal sign segmentation. To the best of our knowledge, this is the first unsupervised method in the domain of temporal sign segmentation.•We experimentally show that there is no single metric dominantly indicating the performance of temporal sign segmentation. Therefore, we introduced a new main metric, the adjusted MF1B, designed by refining the existing metrics to ensure fair and accurate comparison.•We show that CDNC under no supervision achieves a performance comparable to that of the latest semi-supervised method in both a set of existing metrics and the new main metric.