RANK: AI-assisted End-to-End Architecture for Detecting Persistent Attacks in Enterprise Networks

Advanced Persistent Threats (APTs) are sophisticated multi-step attacks, planned and executed by skilled adversaries targeting modern government and enterprise networks. Intrusion Detection Systems (IDSs) and User and Entity Behavior Analytics (UEBA) are commonly employed to aid a security analyst in the detection of APTs. The prolonged nature of APTs, combined with the granular focus of UEBA and IDS, results in overwhelming the analyst with an increasingly impractical number of alerts. Consequent to this abundance of data, and together with the crucial importance of the problem as well as the high cost of the skilled personnel involved, the problem of APT detection becomes a perfect candidate for automation through Artificial Intelligence (AI). In this paper, we provide, up to our knowledge, the first study and implementation of an end-to-end AI-assisted architecture for detecting APTs -- RANK. The goal of the system is not to replace the analyst, rather, it is to automate the complete pipeline from data sources to a final set of incidents for analyst review. The architecture is composed of four consecutive steps: 1) alert templating and merging, 2) alert graph construction, 3) alert graph partitioning into incidents, and 4) incident scoring and ordering. We evaluate our architecture against the 2000 DARPA Intrusion Detection dataset, as well as a read-world private dataset from a medium-scale enterprise. Extensive results are provided showing a three order of magnitude reduction in the amount of data to be reviewed by the analyst, innovative extraction of incidents and security-wise scoring of extracted incidents.