System Log Parsing: A Survey
Modern information and communication systems have become increasingly challenging to manage. The ubiquitous system logs contain plentiful information and are thus widely exploited as an alternative source for system management. As log files usually encompass large amounts of raw data, manually analy...
Gespeichert in:
Veröffentlicht in: | arXiv.org 2022-12 |
---|---|
Hauptverfasser: | , , , , , |
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
container_end_page | |
---|---|
container_issue | |
container_start_page | |
container_title | arXiv.org |
container_volume | |
creator | Zhang, Tianzhu Qiu, Han Castellano, Gabriele Rifai, Myriana Chung Shue Chen Pianese, Fabio |
description | Modern information and communication systems have become increasingly challenging to manage. The ubiquitous system logs contain plentiful information and are thus widely exploited as an alternative source for system management. As log files usually encompass large amounts of raw data, manually analyzing them is laborious and error-prone. Consequently, many research endeavors have been devoted to automatic log analysis. However, these works typically expect structured input and struggle with the heterogeneous nature of raw system logs. Log parsing closes this gap by converting the unstructured system logs to structured records. Many parsers were proposed during the last decades to accommodate various log analysis applications. However, due to the ample solution space and lack of systematic evaluation, it is not easy for practitioners to find ready-made solutions that fit their needs. This paper aims to provide a comprehensive survey on log parsing. We begin with an exhaustive taxonomy of existing log parsers. Then we empirically analyze the critical performance and operational features for 17 open-source solutions both quantitatively and qualitatively, and whenever applicable discuss the merits of alternative approaches. We also elaborate on future challenges and discuss the relevant research directions. We envision this survey as a helpful resource for system administrators and domain experts to choose the most desirable open-source solution or implement new ones based on application-specific requirements. |
doi_str_mv | 10.48550/arxiv.2212.14277 |
format | Article |
fullrecord | <record><control><sourceid>proquest_arxiv</sourceid><recordid>TN_cdi_arxiv_primary_2212_14277</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2760030234</sourcerecordid><originalsourceid>FETCH-LOGICAL-a957-d3efd74d712706e161fb0fe7d238a6e014f2e1fa96dec0d9c6d2388ab443f0ba3</originalsourceid><addsrcrecordid>eNotj8tqwzAUREWh0JDmAwqFGrq2e3WvHnZ3IfQFhhaSvZAjKTg0cSrFof775tHVLGYYzmHsjkMhSinhycbf9lAgciy4QK2v2AiJeF4KxBs2SWkNAKg0Skkjdj8f0t5vsrpbZV82pna7es6m2byPBz_csutgv5Of_OeYLV5fFrP3vP58-5hN69xWUueOfHBaOM1Rg_Jc8dBA8NohlVZ54CKg58FWyvkluGqpTk1pGyEoQGNpzB4ut2d0s4vtxsbBnBTMWeG4eLwsdrH76X3am3XXx-2RyaBWAARIgv4AZ7NHMg</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2760030234</pqid></control><display><type>article</type><title>System Log Parsing: A Survey</title><source>arXiv.org</source><source>Free E- Journals</source><creator>Zhang, Tianzhu ; Qiu, Han ; Castellano, Gabriele ; Rifai, Myriana ; Chung Shue Chen ; Pianese, Fabio</creator><creatorcontrib>Zhang, Tianzhu ; Qiu, Han ; Castellano, Gabriele ; Rifai, Myriana ; Chung Shue Chen ; Pianese, Fabio</creatorcontrib><description>Modern information and communication systems have become increasingly challenging to manage. The ubiquitous system logs contain plentiful information and are thus widely exploited as an alternative source for system management. As log files usually encompass large amounts of raw data, manually analyzing them is laborious and error-prone. Consequently, many research endeavors have been devoted to automatic log analysis. However, these works typically expect structured input and struggle with the heterogeneous nature of raw system logs. Log parsing closes this gap by converting the unstructured system logs to structured records. Many parsers were proposed during the last decades to accommodate various log analysis applications. However, due to the ample solution space and lack of systematic evaluation, it is not easy for practitioners to find ready-made solutions that fit their needs. This paper aims to provide a comprehensive survey on log parsing. We begin with an exhaustive taxonomy of existing log parsers. Then we empirically analyze the critical performance and operational features for 17 open-source solutions both quantitatively and qualitatively, and whenever applicable discuss the merits of alternative approaches. We also elaborate on future challenges and discuss the relevant research directions. We envision this survey as a helpful resource for system administrators and domain experts to choose the most desirable open-source solution or implement new ones based on application-specific requirements.</description><identifier>EISSN: 2331-8422</identifier><identifier>DOI: 10.48550/arxiv.2212.14277</identifier><language>eng</language><publisher>Ithaca: Cornell University Library, arXiv.org</publisher><subject>Communications systems ; Computer Science - Information Retrieval ; Error analysis ; Parsers ; Solution space ; Taxonomy</subject><ispartof>arXiv.org, 2022-12</ispartof><rights>2022. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><rights>http://creativecommons.org/licenses/by/4.0</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>228,230,780,784,885,27925</link.rule.ids><backlink>$$Uhttps://doi.org/10.48550/arXiv.2212.14277$$DView paper in arXiv$$Hfree_for_read</backlink><backlink>$$Uhttps://doi.org/10.1109/TKDE.2022.3222417$$DView published paper (Access to full text may be restricted)$$Hfree_for_read</backlink></links><search><creatorcontrib>Zhang, Tianzhu</creatorcontrib><creatorcontrib>Qiu, Han</creatorcontrib><creatorcontrib>Castellano, Gabriele</creatorcontrib><creatorcontrib>Rifai, Myriana</creatorcontrib><creatorcontrib>Chung Shue Chen</creatorcontrib><creatorcontrib>Pianese, Fabio</creatorcontrib><title>System Log Parsing: A Survey</title><title>arXiv.org</title><description>Modern information and communication systems have become increasingly challenging to manage. The ubiquitous system logs contain plentiful information and are thus widely exploited as an alternative source for system management. As log files usually encompass large amounts of raw data, manually analyzing them is laborious and error-prone. Consequently, many research endeavors have been devoted to automatic log analysis. However, these works typically expect structured input and struggle with the heterogeneous nature of raw system logs. Log parsing closes this gap by converting the unstructured system logs to structured records. Many parsers were proposed during the last decades to accommodate various log analysis applications. However, due to the ample solution space and lack of systematic evaluation, it is not easy for practitioners to find ready-made solutions that fit their needs. This paper aims to provide a comprehensive survey on log parsing. We begin with an exhaustive taxonomy of existing log parsers. Then we empirically analyze the critical performance and operational features for 17 open-source solutions both quantitatively and qualitatively, and whenever applicable discuss the merits of alternative approaches. We also elaborate on future challenges and discuss the relevant research directions. We envision this survey as a helpful resource for system administrators and domain experts to choose the most desirable open-source solution or implement new ones based on application-specific requirements.</description><subject>Communications systems</subject><subject>Computer Science - Information Retrieval</subject><subject>Error analysis</subject><subject>Parsers</subject><subject>Solution space</subject><subject>Taxonomy</subject><issn>2331-8422</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2022</creationdate><recordtype>article</recordtype><sourceid>ABUWG</sourceid><sourceid>AFKRA</sourceid><sourceid>AZQEC</sourceid><sourceid>BENPR</sourceid><sourceid>CCPQU</sourceid><sourceid>DWQXO</sourceid><sourceid>GOX</sourceid><recordid>eNotj8tqwzAUREWh0JDmAwqFGrq2e3WvHnZ3IfQFhhaSvZAjKTg0cSrFof775tHVLGYYzmHsjkMhSinhycbf9lAgciy4QK2v2AiJeF4KxBs2SWkNAKg0Skkjdj8f0t5vsrpbZV82pna7es6m2byPBz_csutgv5Of_OeYLV5fFrP3vP58-5hN69xWUueOfHBaOM1Rg_Jc8dBA8NohlVZ54CKg58FWyvkluGqpTk1pGyEoQGNpzB4ut2d0s4vtxsbBnBTMWeG4eLwsdrH76X3am3XXx-2RyaBWAARIgv4AZ7NHMg</recordid><startdate>20221229</startdate><enddate>20221229</enddate><creator>Zhang, Tianzhu</creator><creator>Qiu, Han</creator><creator>Castellano, Gabriele</creator><creator>Rifai, Myriana</creator><creator>Chung Shue Chen</creator><creator>Pianese, Fabio</creator><general>Cornell University Library, arXiv.org</general><scope>8FE</scope><scope>8FG</scope><scope>ABJCF</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>HCIFZ</scope><scope>L6V</scope><scope>M7S</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>PTHSS</scope><scope>AKY</scope><scope>GOX</scope></search><sort><creationdate>20221229</creationdate><title>System Log Parsing: A Survey</title><author>Zhang, Tianzhu ; Qiu, Han ; Castellano, Gabriele ; Rifai, Myriana ; Chung Shue Chen ; Pianese, Fabio</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-a957-d3efd74d712706e161fb0fe7d238a6e014f2e1fa96dec0d9c6d2388ab443f0ba3</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2022</creationdate><topic>Communications systems</topic><topic>Computer Science - Information Retrieval</topic><topic>Error analysis</topic><topic>Parsers</topic><topic>Solution space</topic><topic>Taxonomy</topic><toplevel>online_resources</toplevel><creatorcontrib>Zhang, Tianzhu</creatorcontrib><creatorcontrib>Qiu, Han</creatorcontrib><creatorcontrib>Castellano, Gabriele</creatorcontrib><creatorcontrib>Rifai, Myriana</creatorcontrib><creatorcontrib>Chung Shue Chen</creatorcontrib><creatorcontrib>Pianese, Fabio</creatorcontrib><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>Materials Science & Engineering Collection</collection><collection>ProQuest Central (Alumni Edition)</collection><collection>ProQuest Central UK/Ireland</collection><collection>ProQuest Central Essentials</collection><collection>ProQuest Central</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central Korea</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Engineering Collection</collection><collection>Engineering Database</collection><collection>Access via ProQuest (Open Access)</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>Engineering Collection</collection><collection>arXiv Computer Science</collection><collection>arXiv.org</collection><jtitle>arXiv.org</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Zhang, Tianzhu</au><au>Qiu, Han</au><au>Castellano, Gabriele</au><au>Rifai, Myriana</au><au>Chung Shue Chen</au><au>Pianese, Fabio</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>System Log Parsing: A Survey</atitle><jtitle>arXiv.org</jtitle><date>2022-12-29</date><risdate>2022</risdate><eissn>2331-8422</eissn><abstract>Modern information and communication systems have become increasingly challenging to manage. The ubiquitous system logs contain plentiful information and are thus widely exploited as an alternative source for system management. As log files usually encompass large amounts of raw data, manually analyzing them is laborious and error-prone. Consequently, many research endeavors have been devoted to automatic log analysis. However, these works typically expect structured input and struggle with the heterogeneous nature of raw system logs. Log parsing closes this gap by converting the unstructured system logs to structured records. Many parsers were proposed during the last decades to accommodate various log analysis applications. However, due to the ample solution space and lack of systematic evaluation, it is not easy for practitioners to find ready-made solutions that fit their needs. This paper aims to provide a comprehensive survey on log parsing. We begin with an exhaustive taxonomy of existing log parsers. Then we empirically analyze the critical performance and operational features for 17 open-source solutions both quantitatively and qualitatively, and whenever applicable discuss the merits of alternative approaches. We also elaborate on future challenges and discuss the relevant research directions. We envision this survey as a helpful resource for system administrators and domain experts to choose the most desirable open-source solution or implement new ones based on application-specific requirements.</abstract><cop>Ithaca</cop><pub>Cornell University Library, arXiv.org</pub><doi>10.48550/arxiv.2212.14277</doi><oa>free_for_read</oa></addata></record> |
fulltext | fulltext |
identifier | EISSN: 2331-8422 |
ispartof | arXiv.org, 2022-12 |
issn | 2331-8422 |
language | eng |
recordid | cdi_arxiv_primary_2212_14277 |
source | arXiv.org; Free E- Journals |
subjects | Communications systems Computer Science - Information Retrieval Error analysis Parsers Solution space Taxonomy |
title | System Log Parsing: A Survey |
url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-23T06%3A35%3A30IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_arxiv&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=System%20Log%20Parsing:%20A%20Survey&rft.jtitle=arXiv.org&rft.au=Zhang,%20Tianzhu&rft.date=2022-12-29&rft.eissn=2331-8422&rft_id=info:doi/10.48550/arxiv.2212.14277&rft_dat=%3Cproquest_arxiv%3E2760030234%3C/proquest_arxiv%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2760030234&rft_id=info:pmid/&rfr_iscdi=true |