Frequency-time domain masking and digital encryption system for DSM-based fiber-wireless integrated system

•This study proposes a novel method for physical layer encryption in fronthaul.•The method introduces a time–frequency domain multidimensional encryption scheme based on the characteristics of DSM, which is compatible with various modulation formats.•The effectiveness of the method was validated through a 4.6 km D-band photonic-assisted millimeter-wave experiment, demonstrating excellent anti-attack performance and a large key space.•This research has potential applications for encryption in both fronthaul and backhaul networks. Delta-sigma modulation can achieve single-carrier high-order quadrature amplitude modulation (QAM) or orthogonal frequency division multiplexing (OFDM) transmission with a very high signal-to-noise ratio. We design an encryption structure that combines delta-sigma modulation (DSM) and digital encryption. This not only improves the flexibility of the system, but also masks the spectral characteristics of the DSM signal. Based on this structure, we additionally demonstrate a frequency-time domain masking (FTDM) and digital encryption scheme with improved confidentiality. Frequency domain masking is realized by digital encryption after DSM, and time domain masking is realized by using multi-scroll chaos superposition on the QAM constellation. In addition, the conventional multi-scroll chaos masking scheme runs the risk of being forced to attack by the blind separation algorithm due to the insufficient power of the chaotic signal. The fidelity of DSM to the signal allows the chaotic signal to mask the constellation points with a higher power ratio. Greatly improves the reliability of this method. Finally, a chaotic-based FTDM and digital encryption scheme with a key space of 10207 is implemented in a photonics-aided millimeter radio-over-fiber (ROF) system employing DSM, multi-scroll chaos masking, and deoxyribonucleic acid (DNA) digital encryption. The equivalent 1.67 GBaud encrypted-4096QAM signal is successfully transmitted and decrypted over a 4.6 km wireless link in the DSM-based fiber-wireless integrated system.