ANOMALY DETECTION USING SEQUENCES OF SYSTEM CALLS

Systems and methods of detecting a call sequence anomaly in a message-based operating system are provided. A message may be received that indicates a programmatic procedure of an operating system was invoked. The message may include a programmatic procedure identifier, a sender process identifier, and a receiver process identifier. An invocation hash may be generated based on the message. The invocation hash may be translated to a smaller invocation identifier. The invocation identifier may be included in a translated call sequence that comprises invocation identifiers for a series of invocations. Depending on whether the translated call sequence is included in previously generated predetermined call sequences, the translated call sequence may be determined as an anomaly or not an anomaly.