Histogram Matrix: Log File Visualization for Anomaly Detection

In today's IT environments, there is an ever increasing demand for log file analysis solutions. Log files often contain important information about possible incidents, but inspecting the often large amounts of textual data is too time-consuming and tedious a task to perform manually. To address this issue, we propose a novel log file visualization technique called Histogram Matrix (HMAT). HMAT visualizes the content of a log file in order to enable a security administrator to efficiently spot anomalies. The system uses a combination of graphical and statistical techniques and allows even non-experts to interactively search for anomalous log messages. Contrary to other approaches, our proposal does not only work on certain special kinds of log files, but instead works on almost every textual log file. Additionally, the system allows to automatically generate security events if an anomaly is detected, similar to anomaly-based intrusion detection systems. This paper introduces HMAT, demonstrates its functionality using log files from a variety of services in real environments, and identifies strengths and limitations of the technique.