Best practices to evaluate the impact of biomedical research software—metric collection beyond citations
Abstract Motivation Software is vital for the advancement of biology and medicine. Impact evaluations of scientific software have primarily emphasized traditional citation metrics of associated papers, despite these metrics inadequately capturing the dynamic picture of impact and despite challenges...
Gespeichert in:
| Veröffentlicht in: | Bioinformatics (Oxford, England) England), 2024-08, Vol.40 (8) |
|---|---|
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , , , |
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | |
|---|---|
| container_issue | 8 |
| container_start_page | |
| container_title | Bioinformatics (Oxford, England) |
| container_volume | 40 |
| creator | Afiaz, Awan Ivanov, Andrey A Chamberlin, John Hanauer, David Savonen, Candace L Goldman, Mary J Morgan, Martin Reich, Michael Getka, Alexander Holmes, Aaron Pati, Sarthak Knight, Dan Boutros, Paul C Bakas, Spyridon Caporaso, J Gregory Del Fiol, Guilherme Hochheiser, Harry Haas, Brian Schloss, Patrick D Eddy, James A Albrecht, Jake Fedorov, Andrey Waldron, Levi Hoffman, Ava M Bradshaw, Richard L Leek, Jeffrey T Wright, Carrie |
| description | Abstract
Motivation
Software is vital for the advancement of biology and medicine. Impact evaluations of scientific software have primarily emphasized traditional citation metrics of associated papers, despite these metrics inadequately capturing the dynamic picture of impact and despite challenges with improper citation.
Results
To understand how software developers evaluate their tools, we conducted a survey of participants in the Informatics Technology for Cancer Research (ITCR) program funded by the National Cancer Institute (NCI). We found that although developers realize the value of more extensive metric collection, they find a lack of funding and time hindering. We also investigated software among this community for how often infrastructure that supports more nontraditional metrics were implemented and how this impacted rates of papers describing usage of the software. We found that infrastructure such as social media presence, more in-depth documentation, the presence of software health metrics, and clear information on how to contact developers seemed to be associated with increased mention rates. Analysing more diverse metrics can enable developers to better understand user engagement, justify continued funding, identify novel use cases, pinpoint improvement areas, and ultimately amplify their software’s impact. Challenges are associated, including distorted or misleading metrics, as well as ethical and security concerns. More attention to nuances involved in capturing impact across the spectrum of biomedical software is needed. For funders and developers, we outline guidance based on experience from our community. By considering how we evaluate software, we can empower developers to create tools that more effectively accelerate biological and medical research progress.
Availability and implementation
More information about the analysis, as well as access to data and code is available at https://github.com/fhdsl/ITCR_Metrics_manuscript_website. |
| doi_str_mv | 10.1093/bioinformatics/btae469 |
| format | Article |
| fullrecord | <record><control><sourceid>proquest_pubme</sourceid><recordid>TN_cdi_pubmedcentral_primary_oai_pubmedcentral_nih_gov_11297485</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><oup_id>10.1093/bioinformatics/btae469</oup_id><sourcerecordid>3124439682</sourcerecordid><originalsourceid>FETCH-LOGICAL-c314t-a338a0085d6b6c7023a3973f69b78ac115083c71644a1e8cadf216d6061cbcd53</originalsourceid><addsrcrecordid>eNqNkc9u1DAQxiMEoqXwCpUlLlyWemLHsU8IKv5JlbjA2Zo4E9arJA6206o3HoIn5Enwapeq5cTJtuY333zjr6rOgb8GbsRF54OfhxAnzN6liy4jSWUeVacgVLuRGuDxvftJ9SylHee84Y16Wp0Iw1XLoT2tdu8oZbZEdEWHEsuB0TWOK2ZieUvMT0spsTCwMnGi3jscWaREGN2WpTDkG4z0--eviXL0jrkwjlS0wsw6ug1zz5zPuH-n59WTAcdEL47nWfXtw_uvl582V18-fr58e7VxAmTeoBAaOddNrzrlWl4LFKYVgzJdq9EBNFwL14KSEoG0w36oQfWKK3Cd6xtxVr056C5rVxw7mnPE0S7RTxhvbUBvH1Zmv7Xfw7UFqE0r9V7h1VEhhh9r-SA7-eRoHHGmsCYrijulhZGmoC__QXdhjXPZzwqopRRG6bpQ6kC5GFKKNNy5AW73edqHedpjnqXx_P4ud21_AywAHICwLv8r-gdwZrb5</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>3124439682</pqid></control><display><type>article</type><title>Best practices to evaluate the impact of biomedical research software—metric collection beyond citations</title><source>Oxford Journals Open Access Collection</source><source>MEDLINE</source><source>DOAJ Directory of Open Access Journals</source><source>Elektronische Zeitschriftenbibliothek - Frei zugängliche E-Journals</source><source>PubMed Central</source><source>Alma/SFX Local Collection</source><creator>Afiaz, Awan ; Ivanov, Andrey A ; Chamberlin, John ; Hanauer, David ; Savonen, Candace L ; Goldman, Mary J ; Morgan, Martin ; Reich, Michael ; Getka, Alexander ; Holmes, Aaron ; Pati, Sarthak ; Knight, Dan ; Boutros, Paul C ; Bakas, Spyridon ; Caporaso, J Gregory ; Del Fiol, Guilherme ; Hochheiser, Harry ; Haas, Brian ; Schloss, Patrick D ; Eddy, James A ; Albrecht, Jake ; Fedorov, Andrey ; Waldron, Levi ; Hoffman, Ava M ; Bradshaw, Richard L ; Leek, Jeffrey T ; Wright, Carrie</creator><creatorcontrib>Afiaz, Awan ; Ivanov, Andrey A ; Chamberlin, John ; Hanauer, David ; Savonen, Candace L ; Goldman, Mary J ; Morgan, Martin ; Reich, Michael ; Getka, Alexander ; Holmes, Aaron ; Pati, Sarthak ; Knight, Dan ; Boutros, Paul C ; Bakas, Spyridon ; Caporaso, J Gregory ; Del Fiol, Guilherme ; Hochheiser, Harry ; Haas, Brian ; Schloss, Patrick D ; Eddy, James A ; Albrecht, Jake ; Fedorov, Andrey ; Waldron, Levi ; Hoffman, Ava M ; Bradshaw, Richard L ; Leek, Jeffrey T ; Wright, Carrie</creatorcontrib><description>Abstract
Motivation
Software is vital for the advancement of biology and medicine. Impact evaluations of scientific software have primarily emphasized traditional citation metrics of associated papers, despite these metrics inadequately capturing the dynamic picture of impact and despite challenges with improper citation.
Results
To understand how software developers evaluate their tools, we conducted a survey of participants in the Informatics Technology for Cancer Research (ITCR) program funded by the National Cancer Institute (NCI). We found that although developers realize the value of more extensive metric collection, they find a lack of funding and time hindering. We also investigated software among this community for how often infrastructure that supports more nontraditional metrics were implemented and how this impacted rates of papers describing usage of the software. We found that infrastructure such as social media presence, more in-depth documentation, the presence of software health metrics, and clear information on how to contact developers seemed to be associated with increased mention rates. Analysing more diverse metrics can enable developers to better understand user engagement, justify continued funding, identify novel use cases, pinpoint improvement areas, and ultimately amplify their software’s impact. Challenges are associated, including distorted or misleading metrics, as well as ethical and security concerns. More attention to nuances involved in capturing impact across the spectrum of biomedical software is needed. For funders and developers, we outline guidance based on experience from our community. By considering how we evaluate software, we can empower developers to create tools that more effectively accelerate biological and medical research progress.
Availability and implementation
More information about the analysis, as well as access to data and code is available at https://github.com/fhdsl/ITCR_Metrics_manuscript_website.</description><identifier>ISSN: 1367-4811</identifier><identifier>ISSN: 1367-4803</identifier><identifier>EISSN: 1367-4811</identifier><identifier>DOI: 10.1093/bioinformatics/btae469</identifier><identifier>PMID: 39067017</identifier><language>eng</language><publisher>England: Oxford University Press</publisher><subject>Availability ; Best practice ; Biological effects ; Biomedical research ; Biomedical Research - methods ; Cancer ; Computational Biology - methods ; Funding ; Humans ; Informatics ; Infrastructure ; Medical research ; Original Paper ; Software ; Software development ; Technology assessment ; United States</subject><ispartof>Bioinformatics (Oxford, England), 2024-08, Vol.40 (8)</ispartof><rights>The Author(s) 2024. Published by Oxford University Press. 2024</rights><rights>The Author(s) 2024. Published by Oxford University Press.</rights><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><cites>FETCH-LOGICAL-c314t-a338a0085d6b6c7023a3973f69b78ac115083c71644a1e8cadf216d6061cbcd53</cites><orcidid>0000-0001-8793-9982 ; 0000-0001-9954-6799 ; 0000-0003-0553-7520 ; 0000-0003-1325-6067 ; 0000-0001-5801-9087 ; 0000-0003-2725-0694 ; 0000-0002-1833-4397 ; 0000-0003-2243-8487 ; 0000-0001-7363-0327 ; 0000-0001-7476-1329</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktopdf>$$Uhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC11297485/pdf/$$EPDF$$P50$$Gpubmedcentral$$Hfree_for_read</linktopdf><linktohtml>$$Uhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC11297485/$$EHTML$$P50$$Gpubmedcentral$$Hfree_for_read</linktohtml><link.rule.ids>230,314,723,776,780,860,881,1598,27901,27902,53766,53768</link.rule.ids><backlink>$$Uhttps://www.ncbi.nlm.nih.gov/pubmed/39067017$$D View this record in MEDLINE/PubMed$$Hfree_for_read</backlink></links><search><creatorcontrib>Afiaz, Awan</creatorcontrib><creatorcontrib>Ivanov, Andrey A</creatorcontrib><creatorcontrib>Chamberlin, John</creatorcontrib><creatorcontrib>Hanauer, David</creatorcontrib><creatorcontrib>Savonen, Candace L</creatorcontrib><creatorcontrib>Goldman, Mary J</creatorcontrib><creatorcontrib>Morgan, Martin</creatorcontrib><creatorcontrib>Reich, Michael</creatorcontrib><creatorcontrib>Getka, Alexander</creatorcontrib><creatorcontrib>Holmes, Aaron</creatorcontrib><creatorcontrib>Pati, Sarthak</creatorcontrib><creatorcontrib>Knight, Dan</creatorcontrib><creatorcontrib>Boutros, Paul C</creatorcontrib><creatorcontrib>Bakas, Spyridon</creatorcontrib><creatorcontrib>Caporaso, J Gregory</creatorcontrib><creatorcontrib>Del Fiol, Guilherme</creatorcontrib><creatorcontrib>Hochheiser, Harry</creatorcontrib><creatorcontrib>Haas, Brian</creatorcontrib><creatorcontrib>Schloss, Patrick D</creatorcontrib><creatorcontrib>Eddy, James A</creatorcontrib><creatorcontrib>Albrecht, Jake</creatorcontrib><creatorcontrib>Fedorov, Andrey</creatorcontrib><creatorcontrib>Waldron, Levi</creatorcontrib><creatorcontrib>Hoffman, Ava M</creatorcontrib><creatorcontrib>Bradshaw, Richard L</creatorcontrib><creatorcontrib>Leek, Jeffrey T</creatorcontrib><creatorcontrib>Wright, Carrie</creatorcontrib><title>Best practices to evaluate the impact of biomedical research software—metric collection beyond citations</title><title>Bioinformatics (Oxford, England)</title><addtitle>Bioinformatics</addtitle><description>Abstract
Motivation
Software is vital for the advancement of biology and medicine. Impact evaluations of scientific software have primarily emphasized traditional citation metrics of associated papers, despite these metrics inadequately capturing the dynamic picture of impact and despite challenges with improper citation.
Results
To understand how software developers evaluate their tools, we conducted a survey of participants in the Informatics Technology for Cancer Research (ITCR) program funded by the National Cancer Institute (NCI). We found that although developers realize the value of more extensive metric collection, they find a lack of funding and time hindering. We also investigated software among this community for how often infrastructure that supports more nontraditional metrics were implemented and how this impacted rates of papers describing usage of the software. We found that infrastructure such as social media presence, more in-depth documentation, the presence of software health metrics, and clear information on how to contact developers seemed to be associated with increased mention rates. Analysing more diverse metrics can enable developers to better understand user engagement, justify continued funding, identify novel use cases, pinpoint improvement areas, and ultimately amplify their software’s impact. Challenges are associated, including distorted or misleading metrics, as well as ethical and security concerns. More attention to nuances involved in capturing impact across the spectrum of biomedical software is needed. For funders and developers, we outline guidance based on experience from our community. By considering how we evaluate software, we can empower developers to create tools that more effectively accelerate biological and medical research progress.
Availability and implementation
More information about the analysis, as well as access to data and code is available at https://github.com/fhdsl/ITCR_Metrics_manuscript_website.</description><subject>Availability</subject><subject>Best practice</subject><subject>Biological effects</subject><subject>Biomedical research</subject><subject>Biomedical Research - methods</subject><subject>Cancer</subject><subject>Computational Biology - methods</subject><subject>Funding</subject><subject>Humans</subject><subject>Informatics</subject><subject>Infrastructure</subject><subject>Medical research</subject><subject>Original Paper</subject><subject>Software</subject><subject>Software development</subject><subject>Technology assessment</subject><subject>United States</subject><issn>1367-4811</issn><issn>1367-4803</issn><issn>1367-4811</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><sourceid>TOX</sourceid><sourceid>EIF</sourceid><recordid>eNqNkc9u1DAQxiMEoqXwCpUlLlyWemLHsU8IKv5JlbjA2Zo4E9arJA6206o3HoIn5Enwapeq5cTJtuY333zjr6rOgb8GbsRF54OfhxAnzN6liy4jSWUeVacgVLuRGuDxvftJ9SylHee84Y16Wp0Iw1XLoT2tdu8oZbZEdEWHEsuB0TWOK2ZieUvMT0spsTCwMnGi3jscWaREGN2WpTDkG4z0--eviXL0jrkwjlS0wsw6ug1zz5zPuH-n59WTAcdEL47nWfXtw_uvl582V18-fr58e7VxAmTeoBAaOddNrzrlWl4LFKYVgzJdq9EBNFwL14KSEoG0w36oQfWKK3Cd6xtxVr056C5rVxw7mnPE0S7RTxhvbUBvH1Zmv7Xfw7UFqE0r9V7h1VEhhh9r-SA7-eRoHHGmsCYrijulhZGmoC__QXdhjXPZzwqopRRG6bpQ6kC5GFKKNNy5AW73edqHedpjnqXx_P4ud21_AywAHICwLv8r-gdwZrb5</recordid><startdate>20240802</startdate><enddate>20240802</enddate><creator>Afiaz, Awan</creator><creator>Ivanov, Andrey A</creator><creator>Chamberlin, John</creator><creator>Hanauer, David</creator><creator>Savonen, Candace L</creator><creator>Goldman, Mary J</creator><creator>Morgan, Martin</creator><creator>Reich, Michael</creator><creator>Getka, Alexander</creator><creator>Holmes, Aaron</creator><creator>Pati, Sarthak</creator><creator>Knight, Dan</creator><creator>Boutros, Paul C</creator><creator>Bakas, Spyridon</creator><creator>Caporaso, J Gregory</creator><creator>Del Fiol, Guilherme</creator><creator>Hochheiser, Harry</creator><creator>Haas, Brian</creator><creator>Schloss, Patrick D</creator><creator>Eddy, James A</creator><creator>Albrecht, Jake</creator><creator>Fedorov, Andrey</creator><creator>Waldron, Levi</creator><creator>Hoffman, Ava M</creator><creator>Bradshaw, Richard L</creator><creator>Leek, Jeffrey T</creator><creator>Wright, Carrie</creator><general>Oxford University Press</general><general>Oxford Publishing Limited (England)</general><scope>TOX</scope><scope>CGR</scope><scope>CUY</scope><scope>CVF</scope><scope>ECM</scope><scope>EIF</scope><scope>NPM</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>7QF</scope><scope>7QO</scope><scope>7QQ</scope><scope>7SC</scope><scope>7SE</scope><scope>7SP</scope><scope>7SR</scope><scope>7TA</scope><scope>7TB</scope><scope>7TM</scope><scope>7TO</scope><scope>7U5</scope><scope>8BQ</scope><scope>8FD</scope><scope>F28</scope><scope>FR3</scope><scope>H8D</scope><scope>H8G</scope><scope>H94</scope><scope>JG9</scope><scope>JQ2</scope><scope>K9.</scope><scope>KR7</scope><scope>L7M</scope><scope>L~C</scope><scope>L~D</scope><scope>P64</scope><scope>7X8</scope><scope>5PM</scope><orcidid>https://orcid.org/0000-0001-8793-9982</orcidid><orcidid>https://orcid.org/0000-0001-9954-6799</orcidid><orcidid>https://orcid.org/0000-0003-0553-7520</orcidid><orcidid>https://orcid.org/0000-0003-1325-6067</orcidid><orcidid>https://orcid.org/0000-0001-5801-9087</orcidid><orcidid>https://orcid.org/0000-0003-2725-0694</orcidid><orcidid>https://orcid.org/0000-0002-1833-4397</orcidid><orcidid>https://orcid.org/0000-0003-2243-8487</orcidid><orcidid>https://orcid.org/0000-0001-7363-0327</orcidid><orcidid>https://orcid.org/0000-0001-7476-1329</orcidid></search><sort><creationdate>20240802</creationdate><title>Best practices to evaluate the impact of biomedical research software—metric collection beyond citations</title><author>Afiaz, Awan ; Ivanov, Andrey A ; Chamberlin, John ; Hanauer, David ; Savonen, Candace L ; Goldman, Mary J ; Morgan, Martin ; Reich, Michael ; Getka, Alexander ; Holmes, Aaron ; Pati, Sarthak ; Knight, Dan ; Boutros, Paul C ; Bakas, Spyridon ; Caporaso, J Gregory ; Del Fiol, Guilherme ; Hochheiser, Harry ; Haas, Brian ; Schloss, Patrick D ; Eddy, James A ; Albrecht, Jake ; Fedorov, Andrey ; Waldron, Levi ; Hoffman, Ava M ; Bradshaw, Richard L ; Leek, Jeffrey T ; Wright, Carrie</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c314t-a338a0085d6b6c7023a3973f69b78ac115083c71644a1e8cadf216d6061cbcd53</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><topic>Availability</topic><topic>Best practice</topic><topic>Biological effects</topic><topic>Biomedical research</topic><topic>Biomedical Research - methods</topic><topic>Cancer</topic><topic>Computational Biology - methods</topic><topic>Funding</topic><topic>Humans</topic><topic>Informatics</topic><topic>Infrastructure</topic><topic>Medical research</topic><topic>Original Paper</topic><topic>Software</topic><topic>Software development</topic><topic>Technology assessment</topic><topic>United States</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Afiaz, Awan</creatorcontrib><creatorcontrib>Ivanov, Andrey A</creatorcontrib><creatorcontrib>Chamberlin, John</creatorcontrib><creatorcontrib>Hanauer, David</creatorcontrib><creatorcontrib>Savonen, Candace L</creatorcontrib><creatorcontrib>Goldman, Mary J</creatorcontrib><creatorcontrib>Morgan, Martin</creatorcontrib><creatorcontrib>Reich, Michael</creatorcontrib><creatorcontrib>Getka, Alexander</creatorcontrib><creatorcontrib>Holmes, Aaron</creatorcontrib><creatorcontrib>Pati, Sarthak</creatorcontrib><creatorcontrib>Knight, Dan</creatorcontrib><creatorcontrib>Boutros, Paul C</creatorcontrib><creatorcontrib>Bakas, Spyridon</creatorcontrib><creatorcontrib>Caporaso, J Gregory</creatorcontrib><creatorcontrib>Del Fiol, Guilherme</creatorcontrib><creatorcontrib>Hochheiser, Harry</creatorcontrib><creatorcontrib>Haas, Brian</creatorcontrib><creatorcontrib>Schloss, Patrick D</creatorcontrib><creatorcontrib>Eddy, James A</creatorcontrib><creatorcontrib>Albrecht, Jake</creatorcontrib><creatorcontrib>Fedorov, Andrey</creatorcontrib><creatorcontrib>Waldron, Levi</creatorcontrib><creatorcontrib>Hoffman, Ava M</creatorcontrib><creatorcontrib>Bradshaw, Richard L</creatorcontrib><creatorcontrib>Leek, Jeffrey T</creatorcontrib><creatorcontrib>Wright, Carrie</creatorcontrib><collection>Oxford Journals Open Access Collection</collection><collection>Medline</collection><collection>MEDLINE</collection><collection>MEDLINE (Ovid)</collection><collection>MEDLINE</collection><collection>MEDLINE</collection><collection>PubMed</collection><collection>CrossRef</collection><collection>Aluminium Industry Abstracts</collection><collection>Biotechnology Research Abstracts</collection><collection>Ceramic Abstracts</collection><collection>Computer and Information Systems Abstracts</collection><collection>Corrosion Abstracts</collection><collection>Electronics & Communications Abstracts</collection><collection>Engineered Materials Abstracts</collection><collection>Materials Business File</collection><collection>Mechanical & Transportation Engineering Abstracts</collection><collection>Nucleic Acids Abstracts</collection><collection>Oncogenes and Growth Factors Abstracts</collection><collection>Solid State and Superconductivity Abstracts</collection><collection>METADEX</collection><collection>Technology Research Database</collection><collection>ANTE: Abstracts in New Technology & Engineering</collection><collection>Engineering Research Database</collection><collection>Aerospace Database</collection><collection>Copper Technical Reference Library</collection><collection>AIDS and Cancer Research Abstracts</collection><collection>Materials Research Database</collection><collection>ProQuest Computer Science Collection</collection><collection>ProQuest Health & Medical Complete (Alumni)</collection><collection>Civil Engineering Abstracts</collection><collection>Advanced Technologies Database with Aerospace</collection><collection>Computer and Information Systems Abstracts Academic</collection><collection>Computer and Information Systems Abstracts Professional</collection><collection>Biotechnology and BioEngineering Abstracts</collection><collection>MEDLINE - Academic</collection><collection>PubMed Central (Full Participant titles)</collection><jtitle>Bioinformatics (Oxford, England)</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Afiaz, Awan</au><au>Ivanov, Andrey A</au><au>Chamberlin, John</au><au>Hanauer, David</au><au>Savonen, Candace L</au><au>Goldman, Mary J</au><au>Morgan, Martin</au><au>Reich, Michael</au><au>Getka, Alexander</au><au>Holmes, Aaron</au><au>Pati, Sarthak</au><au>Knight, Dan</au><au>Boutros, Paul C</au><au>Bakas, Spyridon</au><au>Caporaso, J Gregory</au><au>Del Fiol, Guilherme</au><au>Hochheiser, Harry</au><au>Haas, Brian</au><au>Schloss, Patrick D</au><au>Eddy, James A</au><au>Albrecht, Jake</au><au>Fedorov, Andrey</au><au>Waldron, Levi</au><au>Hoffman, Ava M</au><au>Bradshaw, Richard L</au><au>Leek, Jeffrey T</au><au>Wright, Carrie</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Best practices to evaluate the impact of biomedical research software—metric collection beyond citations</atitle><jtitle>Bioinformatics (Oxford, England)</jtitle><addtitle>Bioinformatics</addtitle><date>2024-08-02</date><risdate>2024</risdate><volume>40</volume><issue>8</issue><issn>1367-4811</issn><issn>1367-4803</issn><eissn>1367-4811</eissn><abstract>Abstract
Motivation
Software is vital for the advancement of biology and medicine. Impact evaluations of scientific software have primarily emphasized traditional citation metrics of associated papers, despite these metrics inadequately capturing the dynamic picture of impact and despite challenges with improper citation.
Results
To understand how software developers evaluate their tools, we conducted a survey of participants in the Informatics Technology for Cancer Research (ITCR) program funded by the National Cancer Institute (NCI). We found that although developers realize the value of more extensive metric collection, they find a lack of funding and time hindering. We also investigated software among this community for how often infrastructure that supports more nontraditional metrics were implemented and how this impacted rates of papers describing usage of the software. We found that infrastructure such as social media presence, more in-depth documentation, the presence of software health metrics, and clear information on how to contact developers seemed to be associated with increased mention rates. Analysing more diverse metrics can enable developers to better understand user engagement, justify continued funding, identify novel use cases, pinpoint improvement areas, and ultimately amplify their software’s impact. Challenges are associated, including distorted or misleading metrics, as well as ethical and security concerns. More attention to nuances involved in capturing impact across the spectrum of biomedical software is needed. For funders and developers, we outline guidance based on experience from our community. By considering how we evaluate software, we can empower developers to create tools that more effectively accelerate biological and medical research progress.
Availability and implementation
More information about the analysis, as well as access to data and code is available at https://github.com/fhdsl/ITCR_Metrics_manuscript_website.</abstract><cop>England</cop><pub>Oxford University Press</pub><pmid>39067017</pmid><doi>10.1093/bioinformatics/btae469</doi><orcidid>https://orcid.org/0000-0001-8793-9982</orcidid><orcidid>https://orcid.org/0000-0001-9954-6799</orcidid><orcidid>https://orcid.org/0000-0003-0553-7520</orcidid><orcidid>https://orcid.org/0000-0003-1325-6067</orcidid><orcidid>https://orcid.org/0000-0001-5801-9087</orcidid><orcidid>https://orcid.org/0000-0003-2725-0694</orcidid><orcidid>https://orcid.org/0000-0002-1833-4397</orcidid><orcidid>https://orcid.org/0000-0003-2243-8487</orcidid><orcidid>https://orcid.org/0000-0001-7363-0327</orcidid><orcidid>https://orcid.org/0000-0001-7476-1329</orcidid><oa>free_for_read</oa></addata></record> |
| fulltext | fulltext |
| identifier | ISSN: 1367-4811 |
| ispartof | Bioinformatics (Oxford, England), 2024-08, Vol.40 (8) |
| issn | 1367-4811 1367-4803 1367-4811 |
| language | eng |
| recordid | cdi_pubmedcentral_primary_oai_pubmedcentral_nih_gov_11297485 |
| source | Oxford Journals Open Access Collection; MEDLINE; DOAJ Directory of Open Access Journals; Elektronische Zeitschriftenbibliothek - Frei zugängliche E-Journals; PubMed Central; Alma/SFX Local Collection |
| subjects | Availability Best practice Biological effects Biomedical research Biomedical Research - methods Cancer Computational Biology - methods Funding Humans Informatics Infrastructure Medical research Original Paper Software Software development Technology assessment United States |
| title | Best practices to evaluate the impact of biomedical research software—metric collection beyond citations |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-02-09T02%3A18%3A20IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_pubme&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Best%20practices%20to%20evaluate%20the%20impact%20of%20biomedical%20research%20software%E2%80%94metric%20collection%20beyond%20citations&rft.jtitle=Bioinformatics%20(Oxford,%20England)&rft.au=Afiaz,%20Awan&rft.date=2024-08-02&rft.volume=40&rft.issue=8&rft.issn=1367-4811&rft.eissn=1367-4811&rft_id=info:doi/10.1093/bioinformatics/btae469&rft_dat=%3Cproquest_pubme%3E3124439682%3C/proquest_pubme%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=3124439682&rft_id=info:pmid/39067017&rft_oup_id=10.1093/bioinformatics/btae469&rfr_iscdi=true |