Metrics to guide development of machine learning algorithms for malaria diagnosis

Automated malaria diagnosis is a difficult but high-value target for machine learning (ML), and effective algorithms could save many thousands of children’s lives. However, current ML efforts largely neglect crucial use case constraints and are thus not clinically useful. Two factors in particular a...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Frontiers in malaria 2024-04, Vol.2
Hauptverfasser: Delahunt, Charles B., Gachuhi, Noni, Horning, Matthew P.
Format: Artikel
Sprache:eng
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page
container_issue
container_start_page
container_title Frontiers in malaria
container_volume 2
creator Delahunt, Charles B.
Gachuhi, Noni
Horning, Matthew P.
description Automated malaria diagnosis is a difficult but high-value target for machine learning (ML), and effective algorithms could save many thousands of children’s lives. However, current ML efforts largely neglect crucial use case constraints and are thus not clinically useful. Two factors in particular are crucial to developing algorithms translatable to clinical field settings: (i) clear understanding of the clinical needs that ML solutions must accommodate; and (ii) task-relevant metrics for guiding and evaluating ML models. Neglect of these factors has seriously hampered past ML work on malaria, because the resulting algorithms do not align with clinical needs. In this paper we address these two issues in the context of automated malaria diagnosis via microscopy on Giemsa-stained blood films. The intended audience are ML researchers as well as anyone evaluating the performance of ML models for malaria. First, we describe why domain expertise is crucial to effectively apply ML to malaria, and list technical documents and other resources that provide this domain knowledge. Second, we detail performance metrics tailored to the clinical requirements of malaria diagnosis, to guide development of ML models and evaluate model performance through the lens of clinical needs (versus a generic ML lens). We highlight the importance of a patient-level perspective, interpatient variability, false positive rates, limit of detection, and different types of error. We also discuss reasons why ROC curves, AUC, and F1, as commonly used in ML work, are poorly suited to this context. These findings also apply to other diseases involving parasite loads, including neglected tropical diseases (NTDs) such as schistosomiasis.
doi_str_mv 10.3389/fmala.2024.1250220
format Article
fullrecord <record><control><sourceid>crossref</sourceid><recordid>TN_cdi_crossref_primary_10_3389_fmala_2024_1250220</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>10_3389_fmala_2024_1250220</sourcerecordid><originalsourceid>FETCH-LOGICAL-c870-a4e3f0924644045afb7f8b04a5fa9d66faacb105b5da416ac48759b02a1ac9c3</originalsourceid><addsrcrecordid>eNpN0MtKxDAYBeAgCg4z8wKu8gKtf25ts5TBG4yIjPvwN006kbYZkir49lKdhatzVufAR8gNg1KIRt_6EQcsOXBZMq6Ac7ggK94wUdRCV5f_-jXZ5vwBAEIwoZhYkbcXN6dgM50j7T9D52jnvtwQT6ObZho9HdEew-To4DBNYeopDn1MYT6OmfqY6PKdAtIuYD_FHPKGXHkcstuec00OD_fvu6di__r4vLvbF7apoUDphAfNZSUlSIW-rX3TgkTlUXdV5RFty0C1qkPJKrSyqZVugSNDq61YE_63alPMOTlvTimMmL4NA7OomF8Vs6iYs4r4AdtQV-o</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype></control><display><type>article</type><title>Metrics to guide development of machine learning algorithms for malaria diagnosis</title><source>DOAJ Directory of Open Access Journals</source><creator>Delahunt, Charles B. ; Gachuhi, Noni ; Horning, Matthew P.</creator><creatorcontrib>Delahunt, Charles B. ; Gachuhi, Noni ; Horning, Matthew P.</creatorcontrib><description>Automated malaria diagnosis is a difficult but high-value target for machine learning (ML), and effective algorithms could save many thousands of children’s lives. However, current ML efforts largely neglect crucial use case constraints and are thus not clinically useful. Two factors in particular are crucial to developing algorithms translatable to clinical field settings: (i) clear understanding of the clinical needs that ML solutions must accommodate; and (ii) task-relevant metrics for guiding and evaluating ML models. Neglect of these factors has seriously hampered past ML work on malaria, because the resulting algorithms do not align with clinical needs. In this paper we address these two issues in the context of automated malaria diagnosis via microscopy on Giemsa-stained blood films. The intended audience are ML researchers as well as anyone evaluating the performance of ML models for malaria. First, we describe why domain expertise is crucial to effectively apply ML to malaria, and list technical documents and other resources that provide this domain knowledge. Second, we detail performance metrics tailored to the clinical requirements of malaria diagnosis, to guide development of ML models and evaluate model performance through the lens of clinical needs (versus a generic ML lens). We highlight the importance of a patient-level perspective, interpatient variability, false positive rates, limit of detection, and different types of error. We also discuss reasons why ROC curves, AUC, and F1, as commonly used in ML work, are poorly suited to this context. These findings also apply to other diseases involving parasite loads, including neglected tropical diseases (NTDs) such as schistosomiasis.</description><identifier>ISSN: 2813-7396</identifier><identifier>EISSN: 2813-7396</identifier><identifier>DOI: 10.3389/fmala.2024.1250220</identifier><language>eng</language><ispartof>Frontiers in malaria, 2024-04, Vol.2</ispartof><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><cites>FETCH-LOGICAL-c870-a4e3f0924644045afb7f8b04a5fa9d66faacb105b5da416ac48759b02a1ac9c3</cites></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>314,776,780,860,27903,27904</link.rule.ids></links><search><creatorcontrib>Delahunt, Charles B.</creatorcontrib><creatorcontrib>Gachuhi, Noni</creatorcontrib><creatorcontrib>Horning, Matthew P.</creatorcontrib><title>Metrics to guide development of machine learning algorithms for malaria diagnosis</title><title>Frontiers in malaria</title><description>Automated malaria diagnosis is a difficult but high-value target for machine learning (ML), and effective algorithms could save many thousands of children’s lives. However, current ML efforts largely neglect crucial use case constraints and are thus not clinically useful. Two factors in particular are crucial to developing algorithms translatable to clinical field settings: (i) clear understanding of the clinical needs that ML solutions must accommodate; and (ii) task-relevant metrics for guiding and evaluating ML models. Neglect of these factors has seriously hampered past ML work on malaria, because the resulting algorithms do not align with clinical needs. In this paper we address these two issues in the context of automated malaria diagnosis via microscopy on Giemsa-stained blood films. The intended audience are ML researchers as well as anyone evaluating the performance of ML models for malaria. First, we describe why domain expertise is crucial to effectively apply ML to malaria, and list technical documents and other resources that provide this domain knowledge. Second, we detail performance metrics tailored to the clinical requirements of malaria diagnosis, to guide development of ML models and evaluate model performance through the lens of clinical needs (versus a generic ML lens). We highlight the importance of a patient-level perspective, interpatient variability, false positive rates, limit of detection, and different types of error. We also discuss reasons why ROC curves, AUC, and F1, as commonly used in ML work, are poorly suited to this context. These findings also apply to other diseases involving parasite loads, including neglected tropical diseases (NTDs) such as schistosomiasis.</description><issn>2813-7396</issn><issn>2813-7396</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><recordid>eNpN0MtKxDAYBeAgCg4z8wKu8gKtf25ts5TBG4yIjPvwN006kbYZkir49lKdhatzVufAR8gNg1KIRt_6EQcsOXBZMq6Ac7ggK94wUdRCV5f_-jXZ5vwBAEIwoZhYkbcXN6dgM50j7T9D52jnvtwQT6ObZho9HdEew-To4DBNYeopDn1MYT6OmfqY6PKdAtIuYD_FHPKGXHkcstuec00OD_fvu6di__r4vLvbF7apoUDphAfNZSUlSIW-rX3TgkTlUXdV5RFty0C1qkPJKrSyqZVugSNDq61YE_63alPMOTlvTimMmL4NA7OomF8Vs6iYs4r4AdtQV-o</recordid><startdate>20240417</startdate><enddate>20240417</enddate><creator>Delahunt, Charles B.</creator><creator>Gachuhi, Noni</creator><creator>Horning, Matthew P.</creator><scope>AAYXX</scope><scope>CITATION</scope></search><sort><creationdate>20240417</creationdate><title>Metrics to guide development of machine learning algorithms for malaria diagnosis</title><author>Delahunt, Charles B. ; Gachuhi, Noni ; Horning, Matthew P.</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c870-a4e3f0924644045afb7f8b04a5fa9d66faacb105b5da416ac48759b02a1ac9c3</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Delahunt, Charles B.</creatorcontrib><creatorcontrib>Gachuhi, Noni</creatorcontrib><creatorcontrib>Horning, Matthew P.</creatorcontrib><collection>CrossRef</collection><jtitle>Frontiers in malaria</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Delahunt, Charles B.</au><au>Gachuhi, Noni</au><au>Horning, Matthew P.</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Metrics to guide development of machine learning algorithms for malaria diagnosis</atitle><jtitle>Frontiers in malaria</jtitle><date>2024-04-17</date><risdate>2024</risdate><volume>2</volume><issn>2813-7396</issn><eissn>2813-7396</eissn><abstract>Automated malaria diagnosis is a difficult but high-value target for machine learning (ML), and effective algorithms could save many thousands of children’s lives. However, current ML efforts largely neglect crucial use case constraints and are thus not clinically useful. Two factors in particular are crucial to developing algorithms translatable to clinical field settings: (i) clear understanding of the clinical needs that ML solutions must accommodate; and (ii) task-relevant metrics for guiding and evaluating ML models. Neglect of these factors has seriously hampered past ML work on malaria, because the resulting algorithms do not align with clinical needs. In this paper we address these two issues in the context of automated malaria diagnosis via microscopy on Giemsa-stained blood films. The intended audience are ML researchers as well as anyone evaluating the performance of ML models for malaria. First, we describe why domain expertise is crucial to effectively apply ML to malaria, and list technical documents and other resources that provide this domain knowledge. Second, we detail performance metrics tailored to the clinical requirements of malaria diagnosis, to guide development of ML models and evaluate model performance through the lens of clinical needs (versus a generic ML lens). We highlight the importance of a patient-level perspective, interpatient variability, false positive rates, limit of detection, and different types of error. We also discuss reasons why ROC curves, AUC, and F1, as commonly used in ML work, are poorly suited to this context. These findings also apply to other diseases involving parasite loads, including neglected tropical diseases (NTDs) such as schistosomiasis.</abstract><doi>10.3389/fmala.2024.1250220</doi><oa>free_for_read</oa></addata></record>
fulltext fulltext
identifier ISSN: 2813-7396
ispartof Frontiers in malaria, 2024-04, Vol.2
issn 2813-7396
2813-7396
language eng
recordid cdi_crossref_primary_10_3389_fmala_2024_1250220
source DOAJ Directory of Open Access Journals
title Metrics to guide development of machine learning algorithms for malaria diagnosis
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-24T18%3A21%3A19IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-crossref&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Metrics%20to%20guide%20development%20of%20machine%20learning%20algorithms%20for%20malaria%20diagnosis&rft.jtitle=Frontiers%20in%20malaria&rft.au=Delahunt,%20Charles%20B.&rft.date=2024-04-17&rft.volume=2&rft.issn=2813-7396&rft.eissn=2813-7396&rft_id=info:doi/10.3389/fmala.2024.1250220&rft_dat=%3Ccrossref%3E10_3389_fmala_2024_1250220%3C/crossref%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true