Performance Comparison of Probabilistically Shaped QAM Formats and Hybrid Shaped APSK Formats With Coded Modulation

We study and compare performance of probabilistically shaped (PS)-64QAM formats (with spectral efficiency ranging from 3.6 to 8.6 b/s/Hz) and coded modulation (CM)-64/16APSK formats with PS or geometric shaping (GS) or hybrid shaping (HS) with spectral efficiency ranging from 4.8 to 8.8 b/s/Hz. We show that a well-designed CM-64/16APSK format could exceed the performance of PS-64QAM formats by a small margin both in back-to-back and transmission. This was verified for transmission distances from 4,435 to 22,176 km using laboratory hardware and our implemented FEC. We show similar nonlinearity compensation (NLC) benefit for both PS and CM formats using 3-stage NLC schemes. The NLC benefit increases from 1.2 to 1.6 dB as the transmission distance increases from 4,435 to 9,504 km and then nearly saturates at ∼1.6 dB after ∼10,000 km. Furthermore, we experimentally demonstrate 71.6 Tb/s (7.36 b/s/Hz) using a CM-64/16APSK format and 70.3 Tb/s (7.22 b/s/Hz) with a PS-64QAM format over 6,970 km using C+L band EDFA testbed.