A three-way decision making approach to malware analysis using probabilistic rough sets

•We employ three-way decisions approach to malware analysis using probabilistic rough sets.•Architecture for malware analysis based on three-way decisions is proposed.•Experimental results on UNM dataset advocates for the use of three-way decisions in malware analysis. Malware analysis aims to identify malware by examining applications behaviour on the host operating system. A common issue in malware analysis is how to mitigate and handle the false decisions such as false positives. Existing approaches which are based on two-way decisions (such as acceptance and rejection) for classifying applications behaviour result in two shortcomings. Firstly, the two-way decisions are rigid and strict in the sense that they demand that a classification decision must be made irrespective of the quality of available information. This potentially leads to wrong classification decisions whenever we do not have sufficient and complete information. Secondly, two-way decisions do not involve any explicit mechanism for dealing with the false decisions at the model level. The existing approaches generally work like an add-on to learning models and are only exercised after incorrect decisions are being made by the learning models. This results in additional processing and increases the complexity of the task. In this paper, we investigate a three-way decision making approach based on decisions of acceptance, rejection or deferment. The added deferment decision option provides flexibility for delaying a certain decision whenever we do not have sufficient information. Moreover, it aims to mitigate the false decisions at the model level by determining a tradeoff between different properties of decision making such as accuracy, generality and uncertainty. We considered three-way decisions based on two probabilistic rough set models, namely, game-theoretic rough sets (GTRS) and information-theoretic rough sets (ITRS) in this study. An architecture of malware analysis realized with probabilistic rough sets based three-way decisions is proposed. A new algorithm termed as sequentially stackable linux security (SSLS) based on the proposed architecture is presented. Experimental results on the system call sequences from the UNM data set advocate for the use of three-way decisions in malware analysis.