Designing efficient streaming-data processing for intrusion avoidance and detection engines using entity selection and entity attribute approach

To determine the fastest way of information retrieval from the given stream of datasets inputted into the intrusion observation and avoidance system. Information retrieval for any given intrusion detection system involves complex process of handling large scale data and its metadata and its inference. Therefore, the retrieval of needed factors from input request/message like event type, event attribute, event label etc are of higher order importance. This inference obtained from input data streams, plays a vital role in sophisticated approach of classification and further processing of such events and its categorisation as needed for downstream consumption for any given AI/ML engines which is constantly running to get the insights out of the input‟s streams. The proposed current approach basically makes improvement to the large-scale data handling by making changes to the event stream data by categorical distribution and partitioning of the data by a proper mathematical model to come to balanced way of handling lesser skewness and avoiding multiple iterative processing of the duplicate events of same nature. Often this leads to minimum of 20% lesser time consumption on large scale implementation of message to event and its attribute and entity extraction. In this way we can focus much on large goal of handling inputs for given ML/AI engine to consume better insights and decision making.