Transmission System and Method Using Mixed Modulation Formats

A system and method for transmitting a wavelength division multiplexed (WDM) signal on an optical transmission path. The system includes at least one first modulation format transmitter configured to generate an associated first modulation format signal on an associated signal wavelength using a first modulation format having a first spectral efficiency, and at least one second modulation format transmitter configured to generate an associated second modulation format signal on an associated signal wavelength using a second modulation format having a second spectral efficiency higher than the first spectral efficiency. The second modulation format signals having an optical power set nominally higher than the optical power of the first modulation format signals. The first and second modulation format signals are combined into an aggregate output signal on the optical transmission path.