A novel image encryption scheme based on chaotic apertured fractional Mellin transform and its filter bank

•Proposing chaotic apertured FrMT and its filter bank (CAFrMT-FB).•Introducing chaotic apertures into FrMTs to form CAFrMT-FB.•A novel image encryption scheme based on CAFrMT-FB has been proposed.•The maximum sensitivity of fractional order p can reach 10−16.•The maximum sensitivity of wavelength λ can reach 10−22. A novel image encryption scheme based on chaotic apertured fractional Mellin transform and its filter bank has been proposed in this paper. A nonlinear chaotic apertured fractional Mellin transform (ACFrMT) and its filter bank (ACFrMT-FB) are proposed, where a pair of complementary chaotic apertures is introduced into FrMT to form filter bank and a special algorithm is employed to solve the problem of data redundancy in filter banks. In order to improve the sensitivity of chaotic aperture, two random phases based on chaotic aperture (RPCA) are generated as phase masks. The proposed encryption algorithm not only provides huge key space, but also greatly improves the sensitivities of fractional order p and wavelength λ. The maximum values of those sensitivity can reach 10-16 and 10-22. The experiment results prove that the proposed encryption scheme is efficient and secure. The results are: sensitivity attack analysis (> 94.77%), differential attack analysis (average NPCR =99.6120% and average UACI = 33.4888%), entropy values (average value 7.99928), correlation coefficient values (almost 0), and cipher key space analysis (at least 1.677×2341).