Annotating general objects in video

A method for annotating general objects contained in video content is provided. The method sends video data to a client device and receives a first annotation from the client device defining a boundary around a portion of a first frame of the video data. Then, the first annotation is tracked through multiple frames of the video content. Other annotations determined to be associated with annotation that match the first annotation within a threshold are determined where the other annotations are received from other client devices and located in the first frame or other frames from the first frame. The method combines the other annotations and the first annotation into an object track and associates a tag with the object track. The tag is input by at least one of the client devices.