A new security solution to JPEG using hyper-chaotic system and modified zigzag scan coding

•We discuss the flaws in a paper accepted in your journal which focused on image compression and encryption scheme.•We give a new scheme and overcome the flaws in that paper.•We achieve a better performance in image compression and encryption.•The compression ratio of our scheme is very close to JPEG.•We give more tests to our scheme than that paper. Though JPEG is an excellent compression standard of images, it does not provide any security performance. Thus, a security solution to JPEG was proposed in Zhang et al. (2014). But there are some flaws in Zhang’s scheme and in this paper we propose a new scheme based on discrete hyper-chaotic system and modified zigzag scan coding. By shuffling the identifiers of zigzag scan encoded sequence with hyper-chaotic sequence and accurately encrypting the certain coefficients which have little relationship with the correlation of the plain image in zigzag scan encoded domain, we achieve high compression performance and robust security simultaneously. Meanwhile we present and analyze the flaws in Zhang’s scheme through theoretical analysis and experimental verification, and give the comparisons between our scheme and Zhang’s. Simulation results verify that our method has better performance in security and efficiency.