Content management and control based on tracking of prior content playback behavior

According to one configuration, a content management resource receives feedback indicating playback commands applied to different portions of a given content stream (such as video content) played back by multiple viewers. The feedback can include video playback commands (such as fast forward commands, rewind commands, play commands, etc.) applied by the multiple users to the different portions of the given content stream. The content management resource aggregates the feedback to produce playback control information indicating playback commands applied by users of the multiple playback devices to the different portions of the given content stream. The content management resource uses the playback control information (aggregated feedback) to produce via timing information, locations of desirable segments and/or undesirable segments in the given stream as indicated by the previously applied commands. The content management resource inserts the aggregated feedback into the content metadata. Future user's playback devices are able to utilize the metadata to skip or otherwise control playback of (un)desirable sections of content.