Trick play user activity reconstruction

A system and method for reconstructing trick play user activity are provided and include analyzing playback information of a recorded program during a playback session to identify trick play actions believed to occur during the playback session. The playback information includes a first sequence of separate video offset events associated with different points of time during the playback session at spaced intervals. For each adjacent pair of video offset events separated by one of the spaced intervals, a change of video offset is compared relative to a change of time to determine a type of trick play action occurring between the adjacent pair of video offset events. A sequence of trick play actions is reconstructed from the trick play actions identified. The method may include analyzing additional playback information and merging such information with the reconstructed sequence of trick play actions to further refine the reconstructed sequence.