Unsupervised Detection and Clustering of Malicious TLS Flows
Malware abuses TLS to encrypt its malicious traffic, preventing examination by content signatures and deep packet inspection. Network detection of malicious TLS flows is an important, but challenging, problem. Prior works have proposed supervised machine learning detectors using TLS features. Howeve...
Gespeichert in:
Veröffentlicht in: | arXiv.org 2022-12 |
---|---|
Hauptverfasser: | , , , , |
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
container_end_page | |
---|---|
container_issue | |
container_start_page | |
container_title | arXiv.org |
container_volume | |
creator | Gomez, Gibran Kotzias, Platon Dell'Amico, Matteo Bilge, Leyla Caballero, Juan |
description | Malware abuses TLS to encrypt its malicious traffic, preventing examination by content signatures and deep packet inspection. Network detection of malicious TLS flows is an important, but challenging, problem. Prior works have proposed supervised machine learning detectors using TLS features. However, by trying to represent all malicious traffic, supervised binary detectors produce models that are too loose, thus introducing errors. Furthermore, they do not distinguish flows generated by different malware. On the other hand, supervised multi-class detectors produce tighter models and can classify flows by malware family, but require family labels, which are not available for many samples. To address these limitations, this work proposes a novel unsupervised approach to detect and cluster malicious TLS flows. Our approach takes as input network traces from sandboxes. It clusters similar TLS flows using 90 features that capture properties of the TLS client, TLS server, certificate, and encrypted payload; and uses the clusters to build an unsupervised detector that can assign a malicious flow to the cluster it belongs to, or determine it is benign. We evaluate our approach using 972K traces from a commercial sandbox and 35M TLS flows from a research network. Our clustering shows very high precision and recall with an F1 score of 0.993. We compare our unsupervised detector with two state-of-the-art approaches, showing that it outperforms both. The false detection rate of our detector is 0.032% measured over four months of traffic. |
doi_str_mv | 10.48550/arxiv.2109.03878 |
format | Article |
fullrecord | <record><control><sourceid>proquest_arxiv</sourceid><recordid>TN_cdi_arxiv_primary_2109_03878</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2571340935</sourcerecordid><originalsourceid>FETCH-LOGICAL-a958-d6ca0d46e1da07ce17a0c2bc2ac4800e1a7d3862d56ab3d6dc795199158d92d93</originalsourceid><addsrcrecordid>eNotj09LwzAchoMgOOY-gCcDnjt_SZp_4EWqU6HiwXouWZJJRm1q0k799s7N03t5eHkehC4ILEvFOVyb9B12S0pAL4EpqU7QjDJGClVSeoYWOW8BgApJOWczdPPW52nwaReyd_jOj96OIfbY9A5X3ZRHn0L_juMGP5su2BCnjJv6Fa-6-JXP0enGdNkv_neOmtV9Uz0W9cvDU3VbF0ZzVThhDbhSeOIMSOuJNGDp2lJjSwXgiZGOKUEdF2bNnHBWak60Jlw5TZ1mc3R5vD2ktUMKHyb9tH-J7SFxT1wdiSHFz8nnsd3GKfV7p5ZySVgJmnH2C4gqU2Q</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2571340935</pqid></control><display><type>article</type><title>Unsupervised Detection and Clustering of Malicious TLS Flows</title><source>arXiv.org</source><source>Free E- Journals</source><creator>Gomez, Gibran ; Kotzias, Platon ; Dell'Amico, Matteo ; Bilge, Leyla ; Caballero, Juan</creator><creatorcontrib>Gomez, Gibran ; Kotzias, Platon ; Dell'Amico, Matteo ; Bilge, Leyla ; Caballero, Juan</creatorcontrib><description>Malware abuses TLS to encrypt its malicious traffic, preventing examination by content signatures and deep packet inspection. Network detection of malicious TLS flows is an important, but challenging, problem. Prior works have proposed supervised machine learning detectors using TLS features. However, by trying to represent all malicious traffic, supervised binary detectors produce models that are too loose, thus introducing errors. Furthermore, they do not distinguish flows generated by different malware. On the other hand, supervised multi-class detectors produce tighter models and can classify flows by malware family, but require family labels, which are not available for many samples. To address these limitations, this work proposes a novel unsupervised approach to detect and cluster malicious TLS flows. Our approach takes as input network traces from sandboxes. It clusters similar TLS flows using 90 features that capture properties of the TLS client, TLS server, certificate, and encrypted payload; and uses the clusters to build an unsupervised detector that can assign a malicious flow to the cluster it belongs to, or determine it is benign. We evaluate our approach using 972K traces from a commercial sandbox and 35M TLS flows from a research network. Our clustering shows very high precision and recall with an F1 score of 0.993. We compare our unsupervised detector with two state-of-the-art approaches, showing that it outperforms both. The false detection rate of our detector is 0.032% measured over four months of traffic.</description><identifier>EISSN: 2331-8422</identifier><identifier>DOI: 10.48550/arxiv.2109.03878</identifier><language>eng</language><publisher>Ithaca: Cornell University Library, arXiv.org</publisher><subject>Clustering ; Computer Science - Cryptography and Security ; Detectors ; Inspection ; Machine learning ; Malware ; Sensors</subject><ispartof>arXiv.org, 2022-12</ispartof><rights>2022. This work is published under http://arxiv.org/licenses/nonexclusive-distrib/1.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><rights>http://arxiv.org/licenses/nonexclusive-distrib/1.0</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>228,230,780,784,885,27925</link.rule.ids><backlink>$$Uhttps://doi.org/10.48550/arXiv.2109.03878$$DView paper in arXiv$$Hfree_for_read</backlink><backlink>$$Uhttps://doi.org/10.1155/1970/3676692$$DView published paper (Access to full text may be restricted)$$Hfree_for_read</backlink></links><search><creatorcontrib>Gomez, Gibran</creatorcontrib><creatorcontrib>Kotzias, Platon</creatorcontrib><creatorcontrib>Dell'Amico, Matteo</creatorcontrib><creatorcontrib>Bilge, Leyla</creatorcontrib><creatorcontrib>Caballero, Juan</creatorcontrib><title>Unsupervised Detection and Clustering of Malicious TLS Flows</title><title>arXiv.org</title><description>Malware abuses TLS to encrypt its malicious traffic, preventing examination by content signatures and deep packet inspection. Network detection of malicious TLS flows is an important, but challenging, problem. Prior works have proposed supervised machine learning detectors using TLS features. However, by trying to represent all malicious traffic, supervised binary detectors produce models that are too loose, thus introducing errors. Furthermore, they do not distinguish flows generated by different malware. On the other hand, supervised multi-class detectors produce tighter models and can classify flows by malware family, but require family labels, which are not available for many samples. To address these limitations, this work proposes a novel unsupervised approach to detect and cluster malicious TLS flows. Our approach takes as input network traces from sandboxes. It clusters similar TLS flows using 90 features that capture properties of the TLS client, TLS server, certificate, and encrypted payload; and uses the clusters to build an unsupervised detector that can assign a malicious flow to the cluster it belongs to, or determine it is benign. We evaluate our approach using 972K traces from a commercial sandbox and 35M TLS flows from a research network. Our clustering shows very high precision and recall with an F1 score of 0.993. We compare our unsupervised detector with two state-of-the-art approaches, showing that it outperforms both. The false detection rate of our detector is 0.032% measured over four months of traffic.</description><subject>Clustering</subject><subject>Computer Science - Cryptography and Security</subject><subject>Detectors</subject><subject>Inspection</subject><subject>Machine learning</subject><subject>Malware</subject><subject>Sensors</subject><issn>2331-8422</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2022</creationdate><recordtype>article</recordtype><sourceid>ABUWG</sourceid><sourceid>AFKRA</sourceid><sourceid>AZQEC</sourceid><sourceid>BENPR</sourceid><sourceid>CCPQU</sourceid><sourceid>DWQXO</sourceid><sourceid>GOX</sourceid><recordid>eNotj09LwzAchoMgOOY-gCcDnjt_SZp_4EWqU6HiwXouWZJJRm1q0k799s7N03t5eHkehC4ILEvFOVyb9B12S0pAL4EpqU7QjDJGClVSeoYWOW8BgApJOWczdPPW52nwaReyd_jOj96OIfbY9A5X3ZRHn0L_juMGP5su2BCnjJv6Fa-6-JXP0enGdNkv_neOmtV9Uz0W9cvDU3VbF0ZzVThhDbhSeOIMSOuJNGDp2lJjSwXgiZGOKUEdF2bNnHBWak60Jlw5TZ1mc3R5vD2ktUMKHyb9tH-J7SFxT1wdiSHFz8nnsd3GKfV7p5ZySVgJmnH2C4gqU2Q</recordid><startdate>20221223</startdate><enddate>20221223</enddate><creator>Gomez, Gibran</creator><creator>Kotzias, Platon</creator><creator>Dell'Amico, Matteo</creator><creator>Bilge, Leyla</creator><creator>Caballero, Juan</creator><general>Cornell University Library, arXiv.org</general><scope>8FE</scope><scope>8FG</scope><scope>ABJCF</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>HCIFZ</scope><scope>L6V</scope><scope>M7S</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>PTHSS</scope><scope>AKY</scope><scope>GOX</scope></search><sort><creationdate>20221223</creationdate><title>Unsupervised Detection and Clustering of Malicious TLS Flows</title><author>Gomez, Gibran ; Kotzias, Platon ; Dell'Amico, Matteo ; Bilge, Leyla ; Caballero, Juan</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-a958-d6ca0d46e1da07ce17a0c2bc2ac4800e1a7d3862d56ab3d6dc795199158d92d93</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2022</creationdate><topic>Clustering</topic><topic>Computer Science - Cryptography and Security</topic><topic>Detectors</topic><topic>Inspection</topic><topic>Machine learning</topic><topic>Malware</topic><topic>Sensors</topic><toplevel>online_resources</toplevel><creatorcontrib>Gomez, Gibran</creatorcontrib><creatorcontrib>Kotzias, Platon</creatorcontrib><creatorcontrib>Dell'Amico, Matteo</creatorcontrib><creatorcontrib>Bilge, Leyla</creatorcontrib><creatorcontrib>Caballero, Juan</creatorcontrib><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>Materials Science & Engineering Collection</collection><collection>ProQuest Central (Alumni Edition)</collection><collection>ProQuest Central UK/Ireland</collection><collection>ProQuest Central Essentials</collection><collection>ProQuest Central</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central Korea</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Engineering Collection</collection><collection>Engineering Database</collection><collection>Access via ProQuest (Open Access)</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>Engineering Collection</collection><collection>arXiv Computer Science</collection><collection>arXiv.org</collection><jtitle>arXiv.org</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Gomez, Gibran</au><au>Kotzias, Platon</au><au>Dell'Amico, Matteo</au><au>Bilge, Leyla</au><au>Caballero, Juan</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Unsupervised Detection and Clustering of Malicious TLS Flows</atitle><jtitle>arXiv.org</jtitle><date>2022-12-23</date><risdate>2022</risdate><eissn>2331-8422</eissn><abstract>Malware abuses TLS to encrypt its malicious traffic, preventing examination by content signatures and deep packet inspection. Network detection of malicious TLS flows is an important, but challenging, problem. Prior works have proposed supervised machine learning detectors using TLS features. However, by trying to represent all malicious traffic, supervised binary detectors produce models that are too loose, thus introducing errors. Furthermore, they do not distinguish flows generated by different malware. On the other hand, supervised multi-class detectors produce tighter models and can classify flows by malware family, but require family labels, which are not available for many samples. To address these limitations, this work proposes a novel unsupervised approach to detect and cluster malicious TLS flows. Our approach takes as input network traces from sandboxes. It clusters similar TLS flows using 90 features that capture properties of the TLS client, TLS server, certificate, and encrypted payload; and uses the clusters to build an unsupervised detector that can assign a malicious flow to the cluster it belongs to, or determine it is benign. We evaluate our approach using 972K traces from a commercial sandbox and 35M TLS flows from a research network. Our clustering shows very high precision and recall with an F1 score of 0.993. We compare our unsupervised detector with two state-of-the-art approaches, showing that it outperforms both. The false detection rate of our detector is 0.032% measured over four months of traffic.</abstract><cop>Ithaca</cop><pub>Cornell University Library, arXiv.org</pub><doi>10.48550/arxiv.2109.03878</doi><oa>free_for_read</oa></addata></record> |
fulltext | fulltext |
identifier | EISSN: 2331-8422 |
ispartof | arXiv.org, 2022-12 |
issn | 2331-8422 |
language | eng |
recordid | cdi_arxiv_primary_2109_03878 |
source | arXiv.org; Free E- Journals |
subjects | Clustering Computer Science - Cryptography and Security Detectors Inspection Machine learning Malware Sensors |
title | Unsupervised Detection and Clustering of Malicious TLS Flows |
url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-24T13%3A32%3A00IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_arxiv&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Unsupervised%20Detection%20and%20Clustering%20of%20Malicious%20TLS%20Flows&rft.jtitle=arXiv.org&rft.au=Gomez,%20Gibran&rft.date=2022-12-23&rft.eissn=2331-8422&rft_id=info:doi/10.48550/arxiv.2109.03878&rft_dat=%3Cproquest_arxiv%3E2571340935%3C/proquest_arxiv%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2571340935&rft_id=info:pmid/&rfr_iscdi=true |