Video transmission system with color prediction and method of operation thereof

A video transmission system and the method of operation thereof includes: a base layer encoder unit encodes a first video frame for generating a first encoded video frame and a color reference frame including dynamically partitioning the color space of the first video frame; an enhancement layer encoding unit for generating a subsequent encoded video frame by differentially encoding a resampled color frame reference and a subsequent video frame; a video stream transport accessed by a bit stream multiplex unit for multiplexing the first encoded video frame and the subsequent encoded video frame; a reference capture unit to generate a decoded video stream by capturing the resampled color frame reference, for decoding the video stream transport, by applying the subsequent encoded video frame to the resampled color frame reference; and a display, coupled to the reference capture unit and the decoded video stream from the video stream transport, for displaying the decoded video stream.