Behavior-based ransomware classification: A particle swarm optimization wrapper-based approach for feature selection

Ransomware is malware that encrypts the victim’s data and demands a ransom for a decryption key. The increasing number of ransomware families and their variants renders the existing signature-based anti-ransomware techniques useless; thus, behavior-based detection techniques have gained popularity. A difficulty in behavior-based ransomware detection is that hundreds of thousands of system calls are obtained as analysis output, making the manual investigation and selection of ransomware-specific features infeasible. Moreover, manual investigation of the analysis output requires domain experts, who are expensive to hire and unavailable in some cases. Machine learning methods have shown success in a wide range of scientific domains to automate and address the problem of feature selection and extraction from noisy and high-dimensional data. However, automated feature selection is under-explored in malware detection. This study proposes an automated feature selection method that utilizes particle swarm optimization for behavior-based ransomware detection and classification. The proposed method considers the significance of various feature groups of the data in ransomware detection and classification and performs feature selection based on groups’ significance. The experimental results show that, in most cases, the proposed method achieves comparable or significantly better performance than other state-of-the-art methods used in this study for benchmarking. In addition, this article presents an in-depth analysis of the significance of various features groups and the features selected by the proposed method in ransomware detection and classification. •An automated PSO-based wrapper method for ransomware detection and classification.•No human input required–PSO automatically selects appropriate features group-wise.•The significance of the feature groups to the class labels is analyzed.•A detailed analysis of selected features is provided.