Vigenère scores for malware detection

Previous research has applied classic cryptanalytic techniques to the malware detection problem. Specifically, scores that are based on simple substitution cipher cryptanalysis have been considered. In this research, we analyze two malware scoring techniques based on the classic Vigenère cipher. Our first approach relies only on the index of coincidence (IC), which is used for example, to determine the length of the keyword in a Vigenère ciphertext. To compute the IC, we consider both the Kasisky Test and Friedman’s Test. We also consider a score based on a more complete cryptanalysis of a Vigenère cipher, where the IC calculation is the first step. We find that both of these scores outperform comparable malware scores in selected cases.