Deployment of Real-Time Network Traffic Analysis using GraphBLAS Hypersparse Matrices and D4M Associative Arrays
Matrix/array analysis of networks can provide significant insight into their behavior and aid in their operation and protection. Prior work has demonstrated the analytic, performance, and compression capabilities of GraphBLAS (graphblas.org) hypersparse matrices and D4M (d4m.mit.edu) associative arr...
Gespeichert in:
| Veröffentlicht in: | arXiv.org 2023-12 |
|---|---|
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , |
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | |
|---|---|
| container_issue | |
| container_start_page | |
| container_title | arXiv.org |
| container_volume | |
| creator | Jones, Michael Kepner, Jeremy Prout, Andrew Davis, Timothy Arcand, William Bestor, David Bergeron, William Byun, Chansup Gadepally, Vijay Houle, Micheal Hubbell, Matthew Hayden Jananthan Klein, Anna Milechin, Lauren Morales, Guillermo Mullen, Julie Patel, Ritesh Pisharody, Sandeep Reuther, Albert Rosa, Antonio Samsi, Siddharth Yee, Charles Michaleas, Peter |
| description | Matrix/array analysis of networks can provide significant insight into their behavior and aid in their operation and protection. Prior work has demonstrated the analytic, performance, and compression capabilities of GraphBLAS (graphblas.org) hypersparse matrices and D4M (d4m.mit.edu) associative arrays (a mathematical superset of matrices). Obtaining the benefits of these capabilities requires integrating them into operational systems, which comes with its own unique challenges. This paper describes two examples of real-time operational implementations. First, is an operational GraphBLAS implementation that constructs anonymized hypersparse matrices on a high-bandwidth network tap. Second, is an operational D4M implementation that analyzes daily cloud gateway logs. The architectures of these implementations are presented. Detailed measurements of the resources and the performance are collected and analyzed. The implementations are capable of meeting their operational requirements using modest computational resources (a couple of processing cores). GraphBLAS is well-suited for low-level analysis of high-bandwidth connections with relatively structured network data. D4M is well-suited for higher-level analysis of more unstructured data. This work demonstrates that these technologies can be implemented in operational settings. |
| doi_str_mv | 10.48550/arxiv.2309.02464 |
| format | Article |
| fullrecord | <record><control><sourceid>proquest_arxiv</sourceid><recordid>TN_cdi_arxiv_primary_2309_02464</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2861990050</sourcerecordid><originalsourceid>FETCH-LOGICAL-a954-1f725631efb771c8390523e5a384bdbc1d998df0c1942ecc4173668f78d2a553</originalsourceid><addsrcrecordid>eNotkF1PwjAYhRsTEwnyA7yyidfDfm7t5QQFE9BEuF9eulaLsM12oPv3TvDq3Dw5OedB6IaSsVBSknsIP_44ZpzoMWEiFRdowDiniRKMXaFRjFtCCEszJiUfoGZqm13d7W3V4trhNwu7ZO33Fr_Y9rsOn3gdwDlvcF7Bros-4kP01TueBWg-Hhb5Cs-7xobYQIgWL6EN3tiIoSrxVCxxHmNtPLT-aHEeAnTxGl062EU7-s8hWj09rifzZPE6e57kiwS0FAl1_b6UU-s2WUaN4ppIxq0ErsSm3Bhaaq1KRwzVglljBM14miqXqZJBf2yIbs-tJxtFE_weQlf8WSlOVnri7kw0of462NgW2_oQ-pOxYCqlWhMiCf8FHMJk1Q</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2861990050</pqid></control><display><type>article</type><title>Deployment of Real-Time Network Traffic Analysis using GraphBLAS Hypersparse Matrices and D4M Associative Arrays</title><source>World Web Journals</source><source>arXiv.org</source><creator>Jones, Michael ; Kepner, Jeremy ; Prout, Andrew ; Davis, Timothy ; Arcand, William ; Bestor, David ; Bergeron, William ; Byun, Chansup ; Gadepally, Vijay ; Houle, Micheal ; Hubbell, Matthew ; Hayden Jananthan ; Klein, Anna ; Milechin, Lauren ; Morales, Guillermo ; Mullen, Julie ; Patel, Ritesh ; Pisharody, Sandeep ; Reuther, Albert ; Rosa, Antonio ; Samsi, Siddharth ; Yee, Charles ; Michaleas, Peter</creator><creatorcontrib>Jones, Michael ; Kepner, Jeremy ; Prout, Andrew ; Davis, Timothy ; Arcand, William ; Bestor, David ; Bergeron, William ; Byun, Chansup ; Gadepally, Vijay ; Houle, Micheal ; Hubbell, Matthew ; Hayden Jananthan ; Klein, Anna ; Milechin, Lauren ; Morales, Guillermo ; Mullen, Julie ; Patel, Ritesh ; Pisharody, Sandeep ; Reuther, Albert ; Rosa, Antonio ; Samsi, Siddharth ; Yee, Charles ; Michaleas, Peter</creatorcontrib><description>Matrix/array analysis of networks can provide significant insight into their behavior and aid in their operation and protection. Prior work has demonstrated the analytic, performance, and compression capabilities of GraphBLAS (graphblas.org) hypersparse matrices and D4M (d4m.mit.edu) associative arrays (a mathematical superset of matrices). Obtaining the benefits of these capabilities requires integrating them into operational systems, which comes with its own unique challenges. This paper describes two examples of real-time operational implementations. First, is an operational GraphBLAS implementation that constructs anonymized hypersparse matrices on a high-bandwidth network tap. Second, is an operational D4M implementation that analyzes daily cloud gateway logs. The architectures of these implementations are presented. Detailed measurements of the resources and the performance are collected and analyzed. The implementations are capable of meeting their operational requirements using modest computational resources (a couple of processing cores). GraphBLAS is well-suited for low-level analysis of high-bandwidth connections with relatively structured network data. D4M is well-suited for higher-level analysis of more unstructured data. This work demonstrates that these technologies can be implemented in operational settings.</description><identifier>EISSN: 2331-8422</identifier><identifier>DOI: 10.48550/arxiv.2309.02464</identifier><language>eng</language><publisher>Ithaca: Cornell University Library, arXiv.org</publisher><subject>Arrays ; Bandwidths ; Cloud computing ; Communications traffic ; Computer Science - Networking and Internet Architecture ; Computer Science - Social and Information Networks ; Mathematical analysis ; Network analysis ; Real time operation ; Traffic analysis ; Unstructured data</subject><ispartof>arXiv.org, 2023-12</ispartof><rights>2023. This work is published under http://arxiv.org/licenses/nonexclusive-distrib/1.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><rights>http://arxiv.org/licenses/nonexclusive-distrib/1.0</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>228,230,776,780,881,27902</link.rule.ids><backlink>$$Uhttps://doi.org/10.48550/arXiv.2309.02464$$DView paper in arXiv$$Hfree_for_read</backlink><backlink>$$Uhttps://doi.org/10.1109/HPEC58863.2023.10363581$$DView published paper (Access to full text may be restricted)$$Hfree_for_read</backlink></links><search><creatorcontrib>Jones, Michael</creatorcontrib><creatorcontrib>Kepner, Jeremy</creatorcontrib><creatorcontrib>Prout, Andrew</creatorcontrib><creatorcontrib>Davis, Timothy</creatorcontrib><creatorcontrib>Arcand, William</creatorcontrib><creatorcontrib>Bestor, David</creatorcontrib><creatorcontrib>Bergeron, William</creatorcontrib><creatorcontrib>Byun, Chansup</creatorcontrib><creatorcontrib>Gadepally, Vijay</creatorcontrib><creatorcontrib>Houle, Micheal</creatorcontrib><creatorcontrib>Hubbell, Matthew</creatorcontrib><creatorcontrib>Hayden Jananthan</creatorcontrib><creatorcontrib>Klein, Anna</creatorcontrib><creatorcontrib>Milechin, Lauren</creatorcontrib><creatorcontrib>Morales, Guillermo</creatorcontrib><creatorcontrib>Mullen, Julie</creatorcontrib><creatorcontrib>Patel, Ritesh</creatorcontrib><creatorcontrib>Pisharody, Sandeep</creatorcontrib><creatorcontrib>Reuther, Albert</creatorcontrib><creatorcontrib>Rosa, Antonio</creatorcontrib><creatorcontrib>Samsi, Siddharth</creatorcontrib><creatorcontrib>Yee, Charles</creatorcontrib><creatorcontrib>Michaleas, Peter</creatorcontrib><title>Deployment of Real-Time Network Traffic Analysis using GraphBLAS Hypersparse Matrices and D4M Associative Arrays</title><title>arXiv.org</title><description>Matrix/array analysis of networks can provide significant insight into their behavior and aid in their operation and protection. Prior work has demonstrated the analytic, performance, and compression capabilities of GraphBLAS (graphblas.org) hypersparse matrices and D4M (d4m.mit.edu) associative arrays (a mathematical superset of matrices). Obtaining the benefits of these capabilities requires integrating them into operational systems, which comes with its own unique challenges. This paper describes two examples of real-time operational implementations. First, is an operational GraphBLAS implementation that constructs anonymized hypersparse matrices on a high-bandwidth network tap. Second, is an operational D4M implementation that analyzes daily cloud gateway logs. The architectures of these implementations are presented. Detailed measurements of the resources and the performance are collected and analyzed. The implementations are capable of meeting their operational requirements using modest computational resources (a couple of processing cores). GraphBLAS is well-suited for low-level analysis of high-bandwidth connections with relatively structured network data. D4M is well-suited for higher-level analysis of more unstructured data. This work demonstrates that these technologies can be implemented in operational settings.</description><subject>Arrays</subject><subject>Bandwidths</subject><subject>Cloud computing</subject><subject>Communications traffic</subject><subject>Computer Science - Networking and Internet Architecture</subject><subject>Computer Science - Social and Information Networks</subject><subject>Mathematical analysis</subject><subject>Network analysis</subject><subject>Real time operation</subject><subject>Traffic analysis</subject><subject>Unstructured data</subject><issn>2331-8422</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2023</creationdate><recordtype>article</recordtype><sourceid>BENPR</sourceid><sourceid>GOX</sourceid><recordid>eNotkF1PwjAYhRsTEwnyA7yyidfDfm7t5QQFE9BEuF9eulaLsM12oPv3TvDq3Dw5OedB6IaSsVBSknsIP_44ZpzoMWEiFRdowDiniRKMXaFRjFtCCEszJiUfoGZqm13d7W3V4trhNwu7ZO33Fr_Y9rsOn3gdwDlvcF7Bros-4kP01TueBWg-Hhb5Cs-7xobYQIgWL6EN3tiIoSrxVCxxHmNtPLT-aHEeAnTxGl062EU7-s8hWj09rifzZPE6e57kiwS0FAl1_b6UU-s2WUaN4ppIxq0ErsSm3Bhaaq1KRwzVglljBM14miqXqZJBf2yIbs-tJxtFE_weQlf8WSlOVnri7kw0of462NgW2_oQ-pOxYCqlWhMiCf8FHMJk1Q</recordid><startdate>20231208</startdate><enddate>20231208</enddate><creator>Jones, Michael</creator><creator>Kepner, Jeremy</creator><creator>Prout, Andrew</creator><creator>Davis, Timothy</creator><creator>Arcand, William</creator><creator>Bestor, David</creator><creator>Bergeron, William</creator><creator>Byun, Chansup</creator><creator>Gadepally, Vijay</creator><creator>Houle, Micheal</creator><creator>Hubbell, Matthew</creator><creator>Hayden Jananthan</creator><creator>Klein, Anna</creator><creator>Milechin, Lauren</creator><creator>Morales, Guillermo</creator><creator>Mullen, Julie</creator><creator>Patel, Ritesh</creator><creator>Pisharody, Sandeep</creator><creator>Reuther, Albert</creator><creator>Rosa, Antonio</creator><creator>Samsi, Siddharth</creator><creator>Yee, Charles</creator><creator>Michaleas, Peter</creator><general>Cornell University Library, arXiv.org</general><scope>8FE</scope><scope>8FG</scope><scope>ABJCF</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>HCIFZ</scope><scope>L6V</scope><scope>M7S</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>PTHSS</scope><scope>AKY</scope><scope>GOX</scope></search><sort><creationdate>20231208</creationdate><title>Deployment of Real-Time Network Traffic Analysis using GraphBLAS Hypersparse Matrices and D4M Associative Arrays</title><author>Jones, Michael ; Kepner, Jeremy ; Prout, Andrew ; Davis, Timothy ; Arcand, William ; Bestor, David ; Bergeron, William ; Byun, Chansup ; Gadepally, Vijay ; Houle, Micheal ; Hubbell, Matthew ; Hayden Jananthan ; Klein, Anna ; Milechin, Lauren ; Morales, Guillermo ; Mullen, Julie ; Patel, Ritesh ; Pisharody, Sandeep ; Reuther, Albert ; Rosa, Antonio ; Samsi, Siddharth ; Yee, Charles ; Michaleas, Peter</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-a954-1f725631efb771c8390523e5a384bdbc1d998df0c1942ecc4173668f78d2a553</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2023</creationdate><topic>Arrays</topic><topic>Bandwidths</topic><topic>Cloud computing</topic><topic>Communications traffic</topic><topic>Computer Science - Networking and Internet Architecture</topic><topic>Computer Science - Social and Information Networks</topic><topic>Mathematical analysis</topic><topic>Network analysis</topic><topic>Real time operation</topic><topic>Traffic analysis</topic><topic>Unstructured data</topic><toplevel>online_resources</toplevel><creatorcontrib>Jones, Michael</creatorcontrib><creatorcontrib>Kepner, Jeremy</creatorcontrib><creatorcontrib>Prout, Andrew</creatorcontrib><creatorcontrib>Davis, Timothy</creatorcontrib><creatorcontrib>Arcand, William</creatorcontrib><creatorcontrib>Bestor, David</creatorcontrib><creatorcontrib>Bergeron, William</creatorcontrib><creatorcontrib>Byun, Chansup</creatorcontrib><creatorcontrib>Gadepally, Vijay</creatorcontrib><creatorcontrib>Houle, Micheal</creatorcontrib><creatorcontrib>Hubbell, Matthew</creatorcontrib><creatorcontrib>Hayden Jananthan</creatorcontrib><creatorcontrib>Klein, Anna</creatorcontrib><creatorcontrib>Milechin, Lauren</creatorcontrib><creatorcontrib>Morales, Guillermo</creatorcontrib><creatorcontrib>Mullen, Julie</creatorcontrib><creatorcontrib>Patel, Ritesh</creatorcontrib><creatorcontrib>Pisharody, Sandeep</creatorcontrib><creatorcontrib>Reuther, Albert</creatorcontrib><creatorcontrib>Rosa, Antonio</creatorcontrib><creatorcontrib>Samsi, Siddharth</creatorcontrib><creatorcontrib>Yee, Charles</creatorcontrib><creatorcontrib>Michaleas, Peter</creatorcontrib><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>Materials Science & Engineering Database (Proquest)</collection><collection>ProQuest Central (Alumni)</collection><collection>ProQuest Central UK/Ireland</collection><collection>ProQuest Central Essentials</collection><collection>ProQuest Central</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Engineering Collection</collection><collection>ProQuest Engineering Database</collection><collection>Publicly Available Content (ProQuest)</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>Engineering collection</collection><collection>arXiv Computer Science</collection><collection>arXiv.org</collection><jtitle>arXiv.org</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Jones, Michael</au><au>Kepner, Jeremy</au><au>Prout, Andrew</au><au>Davis, Timothy</au><au>Arcand, William</au><au>Bestor, David</au><au>Bergeron, William</au><au>Byun, Chansup</au><au>Gadepally, Vijay</au><au>Houle, Micheal</au><au>Hubbell, Matthew</au><au>Hayden Jananthan</au><au>Klein, Anna</au><au>Milechin, Lauren</au><au>Morales, Guillermo</au><au>Mullen, Julie</au><au>Patel, Ritesh</au><au>Pisharody, Sandeep</au><au>Reuther, Albert</au><au>Rosa, Antonio</au><au>Samsi, Siddharth</au><au>Yee, Charles</au><au>Michaleas, Peter</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Deployment of Real-Time Network Traffic Analysis using GraphBLAS Hypersparse Matrices and D4M Associative Arrays</atitle><jtitle>arXiv.org</jtitle><date>2023-12-08</date><risdate>2023</risdate><eissn>2331-8422</eissn><abstract>Matrix/array analysis of networks can provide significant insight into their behavior and aid in their operation and protection. Prior work has demonstrated the analytic, performance, and compression capabilities of GraphBLAS (graphblas.org) hypersparse matrices and D4M (d4m.mit.edu) associative arrays (a mathematical superset of matrices). Obtaining the benefits of these capabilities requires integrating them into operational systems, which comes with its own unique challenges. This paper describes two examples of real-time operational implementations. First, is an operational GraphBLAS implementation that constructs anonymized hypersparse matrices on a high-bandwidth network tap. Second, is an operational D4M implementation that analyzes daily cloud gateway logs. The architectures of these implementations are presented. Detailed measurements of the resources and the performance are collected and analyzed. The implementations are capable of meeting their operational requirements using modest computational resources (a couple of processing cores). GraphBLAS is well-suited for low-level analysis of high-bandwidth connections with relatively structured network data. D4M is well-suited for higher-level analysis of more unstructured data. This work demonstrates that these technologies can be implemented in operational settings.</abstract><cop>Ithaca</cop><pub>Cornell University Library, arXiv.org</pub><doi>10.48550/arxiv.2309.02464</doi><oa>free_for_read</oa></addata></record> |
| fulltext | fulltext |
| identifier | EISSN: 2331-8422 |
| ispartof | arXiv.org, 2023-12 |
| issn | 2331-8422 |
| language | eng |
| recordid | cdi_arxiv_primary_2309_02464 |
| source | World Web Journals; arXiv.org |
| subjects | Arrays Bandwidths Cloud computing Communications traffic Computer Science - Networking and Internet Architecture Computer Science - Social and Information Networks Mathematical analysis Network analysis Real time operation Traffic analysis Unstructured data |
| title | Deployment of Real-Time Network Traffic Analysis using GraphBLAS Hypersparse Matrices and D4M Associative Arrays |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-02-11T15%3A30%3A39IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_arxiv&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Deployment%20of%20Real-Time%20Network%20Traffic%20Analysis%20using%20GraphBLAS%20Hypersparse%20Matrices%20and%20D4M%20Associative%20Arrays&rft.jtitle=arXiv.org&rft.au=Jones,%20Michael&rft.date=2023-12-08&rft.eissn=2331-8422&rft_id=info:doi/10.48550/arxiv.2309.02464&rft_dat=%3Cproquest_arxiv%3E2861990050%3C/proquest_arxiv%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2861990050&rft_id=info:pmid/&rfr_iscdi=true |