Cryptanalysis of a colour image encryption using chaotic APFM nonlinear adaptive filter

•An image encryption that can pass the NIST SP 800–22a tests has been broken.•The adjusted value high related to APFM filter can be revealed by histogram.•Two different cryptanalysis methods have been proposed.•The first method owns high generality.•The second method minimizes the number of used plain/ciphered images. Recently, a novel colour image encryption scheme was proposed based on chaotic Amplitude Phase Frequency Model nonlinear adaptive filter. The authors claim that the scheme achieved sufficient security strength by 14 sub-keys with a total effective key length of 651 bits. However, the attacker can recover the encrypted image without using the secret sub-keys. The critical problem of the original cryptosystem is that the sum of the pixels of the plain image (Mod 256) to generate pseudorandom sequences can be determined. This security weakness leads to collapse of the cryptosystem. By chosen-plaintext attack, two efficient cryptanalysis methods are proposed in this paper, where one has high generality and high computational complexity and the other has low complexity and low generality. Experimental results demonstrate that the cryptosystem can be broken by these two cryptanalysis methods with 774 and 265 pairs of chosen plain/cipher images.