Detection of visual tracker divergence

A system comprising at least one hardware processor; and a non-transitory computer-readable storage medium having stored thereon program instructions, the program instructions executable by the at least one hardware processor to: receive, as input, video data, perform a first tracking, using a tracking algorithm, of an identified target in a sequence of frames of said video data, wherein said first tracking is performed forward in time, perform a second tracking, using said tracking algorithm, of said target in said sequence, wherein said second tracking is performed backward in time, and calculate a confidence score for a tracked location of said target in a frame of said sequence, based, at least in part, on a comparison between said first tracking and said second tracking.