Unsupervised Detection and Clustering of Malicious TLS Flows

Malware abuses TLS to encrypt its malicious traffic, preventing examination by content signatures and deep packet inspection. Network detection of malicious TLS flows is an important, but challenging, problem. Prior works have proposed supervised machine learning detectors using TLS features. Howeve...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:arXiv.org 2022-12
Hauptverfasser: Gomez, Gibran, Kotzias, Platon, Dell'Amico, Matteo, Bilge, Leyla, Caballero, Juan
Format: Artikel
Sprache:eng
Schlagworte:
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page
container_issue
container_start_page
container_title arXiv.org
container_volume
creator Gomez, Gibran
Kotzias, Platon
Dell'Amico, Matteo
Bilge, Leyla
Caballero, Juan
description Malware abuses TLS to encrypt its malicious traffic, preventing examination by content signatures and deep packet inspection. Network detection of malicious TLS flows is an important, but challenging, problem. Prior works have proposed supervised machine learning detectors using TLS features. However, by trying to represent all malicious traffic, supervised binary detectors produce models that are too loose, thus introducing errors. Furthermore, they do not distinguish flows generated by different malware. On the other hand, supervised multi-class detectors produce tighter models and can classify flows by malware family, but require family labels, which are not available for many samples. To address these limitations, this work proposes a novel unsupervised approach to detect and cluster malicious TLS flows. Our approach takes as input network traces from sandboxes. It clusters similar TLS flows using 90 features that capture properties of the TLS client, TLS server, certificate, and encrypted payload; and uses the clusters to build an unsupervised detector that can assign a malicious flow to the cluster it belongs to, or determine it is benign. We evaluate our approach using 972K traces from a commercial sandbox and 35M TLS flows from a research network. Our clustering shows very high precision and recall with an F1 score of 0.993. We compare our unsupervised detector with two state-of-the-art approaches, showing that it outperforms both. The false detection rate of our detector is 0.032% measured over four months of traffic.
doi_str_mv 10.48550/arxiv.2109.03878
format Article
fullrecord <record><control><sourceid>proquest_arxiv</sourceid><recordid>TN_cdi_arxiv_primary_2109_03878</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2571340935</sourcerecordid><originalsourceid>FETCH-LOGICAL-a958-d6ca0d46e1da07ce17a0c2bc2ac4800e1a7d3862d56ab3d6dc795199158d92d93</originalsourceid><addsrcrecordid>eNotj09LwzAchoMgOOY-gCcDnjt_SZp_4EWqU6HiwXouWZJJRm1q0k799s7N03t5eHkehC4ILEvFOVyb9B12S0pAL4EpqU7QjDJGClVSeoYWOW8BgApJOWczdPPW52nwaReyd_jOj96OIfbY9A5X3ZRHn0L_juMGP5su2BCnjJv6Fa-6-JXP0enGdNkv_neOmtV9Uz0W9cvDU3VbF0ZzVThhDbhSeOIMSOuJNGDp2lJjSwXgiZGOKUEdF2bNnHBWak60Jlw5TZ1mc3R5vD2ktUMKHyb9tH-J7SFxT1wdiSHFz8nnsd3GKfV7p5ZySVgJmnH2C4gqU2Q</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2571340935</pqid></control><display><type>article</type><title>Unsupervised Detection and Clustering of Malicious TLS Flows</title><source>arXiv.org</source><source>Free E- Journals</source><creator>Gomez, Gibran ; Kotzias, Platon ; Dell'Amico, Matteo ; Bilge, Leyla ; Caballero, Juan</creator><creatorcontrib>Gomez, Gibran ; Kotzias, Platon ; Dell'Amico, Matteo ; Bilge, Leyla ; Caballero, Juan</creatorcontrib><description>Malware abuses TLS to encrypt its malicious traffic, preventing examination by content signatures and deep packet inspection. Network detection of malicious TLS flows is an important, but challenging, problem. Prior works have proposed supervised machine learning detectors using TLS features. However, by trying to represent all malicious traffic, supervised binary detectors produce models that are too loose, thus introducing errors. Furthermore, they do not distinguish flows generated by different malware. On the other hand, supervised multi-class detectors produce tighter models and can classify flows by malware family, but require family labels, which are not available for many samples. To address these limitations, this work proposes a novel unsupervised approach to detect and cluster malicious TLS flows. Our approach takes as input network traces from sandboxes. It clusters similar TLS flows using 90 features that capture properties of the TLS client, TLS server, certificate, and encrypted payload; and uses the clusters to build an unsupervised detector that can assign a malicious flow to the cluster it belongs to, or determine it is benign. We evaluate our approach using 972K traces from a commercial sandbox and 35M TLS flows from a research network. Our clustering shows very high precision and recall with an F1 score of 0.993. We compare our unsupervised detector with two state-of-the-art approaches, showing that it outperforms both. The false detection rate of our detector is 0.032% measured over four months of traffic.</description><identifier>EISSN: 2331-8422</identifier><identifier>DOI: 10.48550/arxiv.2109.03878</identifier><language>eng</language><publisher>Ithaca: Cornell University Library, arXiv.org</publisher><subject>Clustering ; Computer Science - Cryptography and Security ; Detectors ; Inspection ; Machine learning ; Malware ; Sensors</subject><ispartof>arXiv.org, 2022-12</ispartof><rights>2022. This work is published under http://arxiv.org/licenses/nonexclusive-distrib/1.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><rights>http://arxiv.org/licenses/nonexclusive-distrib/1.0</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>228,230,780,784,885,27925</link.rule.ids><backlink>$$Uhttps://doi.org/10.48550/arXiv.2109.03878$$DView paper in arXiv$$Hfree_for_read</backlink><backlink>$$Uhttps://doi.org/10.1155/1970/3676692$$DView published paper (Access to full text may be restricted)$$Hfree_for_read</backlink></links><search><creatorcontrib>Gomez, Gibran</creatorcontrib><creatorcontrib>Kotzias, Platon</creatorcontrib><creatorcontrib>Dell'Amico, Matteo</creatorcontrib><creatorcontrib>Bilge, Leyla</creatorcontrib><creatorcontrib>Caballero, Juan</creatorcontrib><title>Unsupervised Detection and Clustering of Malicious TLS Flows</title><title>arXiv.org</title><description>Malware abuses TLS to encrypt its malicious traffic, preventing examination by content signatures and deep packet inspection. Network detection of malicious TLS flows is an important, but challenging, problem. Prior works have proposed supervised machine learning detectors using TLS features. However, by trying to represent all malicious traffic, supervised binary detectors produce models that are too loose, thus introducing errors. Furthermore, they do not distinguish flows generated by different malware. On the other hand, supervised multi-class detectors produce tighter models and can classify flows by malware family, but require family labels, which are not available for many samples. To address these limitations, this work proposes a novel unsupervised approach to detect and cluster malicious TLS flows. Our approach takes as input network traces from sandboxes. It clusters similar TLS flows using 90 features that capture properties of the TLS client, TLS server, certificate, and encrypted payload; and uses the clusters to build an unsupervised detector that can assign a malicious flow to the cluster it belongs to, or determine it is benign. We evaluate our approach using 972K traces from a commercial sandbox and 35M TLS flows from a research network. Our clustering shows very high precision and recall with an F1 score of 0.993. We compare our unsupervised detector with two state-of-the-art approaches, showing that it outperforms both. The false detection rate of our detector is 0.032% measured over four months of traffic.</description><subject>Clustering</subject><subject>Computer Science - Cryptography and Security</subject><subject>Detectors</subject><subject>Inspection</subject><subject>Machine learning</subject><subject>Malware</subject><subject>Sensors</subject><issn>2331-8422</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2022</creationdate><recordtype>article</recordtype><sourceid>ABUWG</sourceid><sourceid>AFKRA</sourceid><sourceid>AZQEC</sourceid><sourceid>BENPR</sourceid><sourceid>CCPQU</sourceid><sourceid>DWQXO</sourceid><sourceid>GOX</sourceid><recordid>eNotj09LwzAchoMgOOY-gCcDnjt_SZp_4EWqU6HiwXouWZJJRm1q0k799s7N03t5eHkehC4ILEvFOVyb9B12S0pAL4EpqU7QjDJGClVSeoYWOW8BgApJOWczdPPW52nwaReyd_jOj96OIfbY9A5X3ZRHn0L_juMGP5su2BCnjJv6Fa-6-JXP0enGdNkv_neOmtV9Uz0W9cvDU3VbF0ZzVThhDbhSeOIMSOuJNGDp2lJjSwXgiZGOKUEdF2bNnHBWak60Jlw5TZ1mc3R5vD2ktUMKHyb9tH-J7SFxT1wdiSHFz8nnsd3GKfV7p5ZySVgJmnH2C4gqU2Q</recordid><startdate>20221223</startdate><enddate>20221223</enddate><creator>Gomez, Gibran</creator><creator>Kotzias, Platon</creator><creator>Dell'Amico, Matteo</creator><creator>Bilge, Leyla</creator><creator>Caballero, Juan</creator><general>Cornell University Library, arXiv.org</general><scope>8FE</scope><scope>8FG</scope><scope>ABJCF</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>HCIFZ</scope><scope>L6V</scope><scope>M7S</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>PTHSS</scope><scope>AKY</scope><scope>GOX</scope></search><sort><creationdate>20221223</creationdate><title>Unsupervised Detection and Clustering of Malicious TLS Flows</title><author>Gomez, Gibran ; Kotzias, Platon ; Dell'Amico, Matteo ; Bilge, Leyla ; Caballero, Juan</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-a958-d6ca0d46e1da07ce17a0c2bc2ac4800e1a7d3862d56ab3d6dc795199158d92d93</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2022</creationdate><topic>Clustering</topic><topic>Computer Science - Cryptography and Security</topic><topic>Detectors</topic><topic>Inspection</topic><topic>Machine learning</topic><topic>Malware</topic><topic>Sensors</topic><toplevel>online_resources</toplevel><creatorcontrib>Gomez, Gibran</creatorcontrib><creatorcontrib>Kotzias, Platon</creatorcontrib><creatorcontrib>Dell'Amico, Matteo</creatorcontrib><creatorcontrib>Bilge, Leyla</creatorcontrib><creatorcontrib>Caballero, Juan</creatorcontrib><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>Materials Science &amp; Engineering Collection</collection><collection>ProQuest Central (Alumni Edition)</collection><collection>ProQuest Central UK/Ireland</collection><collection>ProQuest Central Essentials</collection><collection>ProQuest Central</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central Korea</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Engineering Collection</collection><collection>Engineering Database</collection><collection>Access via ProQuest (Open Access)</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>Engineering Collection</collection><collection>arXiv Computer Science</collection><collection>arXiv.org</collection><jtitle>arXiv.org</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Gomez, Gibran</au><au>Kotzias, Platon</au><au>Dell'Amico, Matteo</au><au>Bilge, Leyla</au><au>Caballero, Juan</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Unsupervised Detection and Clustering of Malicious TLS Flows</atitle><jtitle>arXiv.org</jtitle><date>2022-12-23</date><risdate>2022</risdate><eissn>2331-8422</eissn><abstract>Malware abuses TLS to encrypt its malicious traffic, preventing examination by content signatures and deep packet inspection. Network detection of malicious TLS flows is an important, but challenging, problem. Prior works have proposed supervised machine learning detectors using TLS features. However, by trying to represent all malicious traffic, supervised binary detectors produce models that are too loose, thus introducing errors. Furthermore, they do not distinguish flows generated by different malware. On the other hand, supervised multi-class detectors produce tighter models and can classify flows by malware family, but require family labels, which are not available for many samples. To address these limitations, this work proposes a novel unsupervised approach to detect and cluster malicious TLS flows. Our approach takes as input network traces from sandboxes. It clusters similar TLS flows using 90 features that capture properties of the TLS client, TLS server, certificate, and encrypted payload; and uses the clusters to build an unsupervised detector that can assign a malicious flow to the cluster it belongs to, or determine it is benign. We evaluate our approach using 972K traces from a commercial sandbox and 35M TLS flows from a research network. Our clustering shows very high precision and recall with an F1 score of 0.993. We compare our unsupervised detector with two state-of-the-art approaches, showing that it outperforms both. The false detection rate of our detector is 0.032% measured over four months of traffic.</abstract><cop>Ithaca</cop><pub>Cornell University Library, arXiv.org</pub><doi>10.48550/arxiv.2109.03878</doi><oa>free_for_read</oa></addata></record>
fulltext fulltext
identifier EISSN: 2331-8422
ispartof arXiv.org, 2022-12
issn 2331-8422
language eng
recordid cdi_arxiv_primary_2109_03878
source arXiv.org; Free E- Journals
subjects Clustering
Computer Science - Cryptography and Security
Detectors
Inspection
Machine learning
Malware
Sensors
title Unsupervised Detection and Clustering of Malicious TLS Flows
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-24T13%3A32%3A00IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_arxiv&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Unsupervised%20Detection%20and%20Clustering%20of%20Malicious%20TLS%20Flows&rft.jtitle=arXiv.org&rft.au=Gomez,%20Gibran&rft.date=2022-12-23&rft.eissn=2331-8422&rft_id=info:doi/10.48550/arxiv.2109.03878&rft_dat=%3Cproquest_arxiv%3E2571340935%3C/proquest_arxiv%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2571340935&rft_id=info:pmid/&rfr_iscdi=true