REFORMS: Reporting Standards for Machine Learning Based Science
Machine learning (ML) methods are proliferating in scientific research. However, the adoption of these methods has been accompanied by failures of validity, reproducibility, and generalizability. These failures can hinder scientific progress, lead to false consensus around invalid claims, and underm...
Gespeichert in:
Veröffentlicht in: | arXiv.org 2023-09 |
---|---|
Hauptverfasser: | , , , , , , , , , , , , , , , , , , |
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
container_end_page | |
---|---|
container_issue | |
container_start_page | |
container_title | arXiv.org |
container_volume | |
creator | Kapoor, Sayash Cantrell, Emily Kenny, Peng Pham, Thanh Hien Bail, Christopher A Gundersen, Odd Erik Hofman, Jake M Hullman, Jessica Lones, Michael A Malik, Momin M Nanayakkara, Priyanka Poldrack, Russell A Inioluwa, Deborah Raji Roberts, Michael Salganik, Matthew J Serra-Garcia, Marta Stewart, Brandon M Vandewiele, Gilles Narayanan, Arvind |
description | Machine learning (ML) methods are proliferating in scientific research. However, the adoption of these methods has been accompanied by failures of validity, reproducibility, and generalizability. These failures can hinder scientific progress, lead to false consensus around invalid claims, and undermine the credibility of ML-based science. ML methods are often applied and fail in similar ways across disciplines. Motivated by this observation, our goal is to provide clear reporting standards for ML-based science. Drawing from an extensive review of past literature, we present the REFORMS checklist (\(\textbf{Re}\)porting Standards \(\textbf{For}\) \(\textbf{M}\)achine Learning Based \(\textbf{S}\)cience). It consists of 32 questions and a paired set of guidelines. REFORMS was developed based on a consensus of 19 researchers across computer science, data science, mathematics, social sciences, and biomedical sciences. REFORMS can serve as a resource for researchers when designing and implementing a study, for referees when reviewing papers, and for journals when enforcing standards for transparency and reproducibility. |
format | Article |
fullrecord | <record><control><sourceid>proquest</sourceid><recordid>TN_cdi_proquest_journals_2851479587</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2851479587</sourcerecordid><originalsourceid>FETCH-proquest_journals_28514795873</originalsourceid><addsrcrecordid>eNpjYuA0MjY21LUwMTLiYOAtLs4yMDAwMjM3MjU15mSwD3J18w_yDbZSCEotyC8qycxLVwguScxLSSxKKVZIyy9S8E1MzsjMS1XwSU0sygNJOyUWp6YoBCdnpuYlp_IwsKYl5hSn8kJpbgZlN9cQZw_dgqL8wtLU4pL4rPzSojygVLyRhamhibmlqYW5MXGqAFZ-N48</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2851479587</pqid></control><display><type>article</type><title>REFORMS: Reporting Standards for Machine Learning Based Science</title><source>Free E- Journals</source><creator>Kapoor, Sayash ; Cantrell, Emily ; Kenny, Peng ; Pham, Thanh Hien ; Bail, Christopher A ; Gundersen, Odd Erik ; Hofman, Jake M ; Hullman, Jessica ; Lones, Michael A ; Malik, Momin M ; Nanayakkara, Priyanka ; Poldrack, Russell A ; Inioluwa, Deborah Raji ; Roberts, Michael ; Salganik, Matthew J ; Serra-Garcia, Marta ; Stewart, Brandon M ; Vandewiele, Gilles ; Narayanan, Arvind</creator><creatorcontrib>Kapoor, Sayash ; Cantrell, Emily ; Kenny, Peng ; Pham, Thanh Hien ; Bail, Christopher A ; Gundersen, Odd Erik ; Hofman, Jake M ; Hullman, Jessica ; Lones, Michael A ; Malik, Momin M ; Nanayakkara, Priyanka ; Poldrack, Russell A ; Inioluwa, Deborah Raji ; Roberts, Michael ; Salganik, Matthew J ; Serra-Garcia, Marta ; Stewart, Brandon M ; Vandewiele, Gilles ; Narayanan, Arvind</creatorcontrib><description>Machine learning (ML) methods are proliferating in scientific research. However, the adoption of these methods has been accompanied by failures of validity, reproducibility, and generalizability. These failures can hinder scientific progress, lead to false consensus around invalid claims, and undermine the credibility of ML-based science. ML methods are often applied and fail in similar ways across disciplines. Motivated by this observation, our goal is to provide clear reporting standards for ML-based science. Drawing from an extensive review of past literature, we present the REFORMS checklist (\(\textbf{Re}\)porting Standards \(\textbf{For}\) \(\textbf{M}\)achine Learning Based \(\textbf{S}\)cience). It consists of 32 questions and a paired set of guidelines. REFORMS was developed based on a consensus of 19 researchers across computer science, data science, mathematics, social sciences, and biomedical sciences. REFORMS can serve as a resource for researchers when designing and implementing a study, for referees when reviewing papers, and for journals when enforcing standards for transparency and reproducibility.</description><identifier>EISSN: 2331-8422</identifier><language>eng</language><publisher>Ithaca: Cornell University Library, arXiv.org</publisher><subject>Literature reviews ; Machine learning ; Reproducibility ; Science</subject><ispartof>arXiv.org, 2023-09</ispartof><rights>2023. This work is published under http://arxiv.org/licenses/nonexclusive-distrib/1.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>780,784</link.rule.ids></links><search><creatorcontrib>Kapoor, Sayash</creatorcontrib><creatorcontrib>Cantrell, Emily</creatorcontrib><creatorcontrib>Kenny, Peng</creatorcontrib><creatorcontrib>Pham, Thanh Hien</creatorcontrib><creatorcontrib>Bail, Christopher A</creatorcontrib><creatorcontrib>Gundersen, Odd Erik</creatorcontrib><creatorcontrib>Hofman, Jake M</creatorcontrib><creatorcontrib>Hullman, Jessica</creatorcontrib><creatorcontrib>Lones, Michael A</creatorcontrib><creatorcontrib>Malik, Momin M</creatorcontrib><creatorcontrib>Nanayakkara, Priyanka</creatorcontrib><creatorcontrib>Poldrack, Russell A</creatorcontrib><creatorcontrib>Inioluwa, Deborah Raji</creatorcontrib><creatorcontrib>Roberts, Michael</creatorcontrib><creatorcontrib>Salganik, Matthew J</creatorcontrib><creatorcontrib>Serra-Garcia, Marta</creatorcontrib><creatorcontrib>Stewart, Brandon M</creatorcontrib><creatorcontrib>Vandewiele, Gilles</creatorcontrib><creatorcontrib>Narayanan, Arvind</creatorcontrib><title>REFORMS: Reporting Standards for Machine Learning Based Science</title><title>arXiv.org</title><description>Machine learning (ML) methods are proliferating in scientific research. However, the adoption of these methods has been accompanied by failures of validity, reproducibility, and generalizability. These failures can hinder scientific progress, lead to false consensus around invalid claims, and undermine the credibility of ML-based science. ML methods are often applied and fail in similar ways across disciplines. Motivated by this observation, our goal is to provide clear reporting standards for ML-based science. Drawing from an extensive review of past literature, we present the REFORMS checklist (\(\textbf{Re}\)porting Standards \(\textbf{For}\) \(\textbf{M}\)achine Learning Based \(\textbf{S}\)cience). It consists of 32 questions and a paired set of guidelines. REFORMS was developed based on a consensus of 19 researchers across computer science, data science, mathematics, social sciences, and biomedical sciences. REFORMS can serve as a resource for researchers when designing and implementing a study, for referees when reviewing papers, and for journals when enforcing standards for transparency and reproducibility.</description><subject>Literature reviews</subject><subject>Machine learning</subject><subject>Reproducibility</subject><subject>Science</subject><issn>2331-8422</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2023</creationdate><recordtype>article</recordtype><sourceid>ABUWG</sourceid><sourceid>AFKRA</sourceid><sourceid>AZQEC</sourceid><sourceid>BENPR</sourceid><sourceid>CCPQU</sourceid><sourceid>DWQXO</sourceid><recordid>eNpjYuA0MjY21LUwMTLiYOAtLs4yMDAwMjM3MjU15mSwD3J18w_yDbZSCEotyC8qycxLVwguScxLSSxKKVZIyy9S8E1MzsjMS1XwSU0sygNJOyUWp6YoBCdnpuYlp_IwsKYl5hSn8kJpbgZlN9cQZw_dgqL8wtLU4pL4rPzSojygVLyRhamhibmlqYW5MXGqAFZ-N48</recordid><startdate>20230919</startdate><enddate>20230919</enddate><creator>Kapoor, Sayash</creator><creator>Cantrell, Emily</creator><creator>Kenny, Peng</creator><creator>Pham, Thanh Hien</creator><creator>Bail, Christopher A</creator><creator>Gundersen, Odd Erik</creator><creator>Hofman, Jake M</creator><creator>Hullman, Jessica</creator><creator>Lones, Michael A</creator><creator>Malik, Momin M</creator><creator>Nanayakkara, Priyanka</creator><creator>Poldrack, Russell A</creator><creator>Inioluwa, Deborah Raji</creator><creator>Roberts, Michael</creator><creator>Salganik, Matthew J</creator><creator>Serra-Garcia, Marta</creator><creator>Stewart, Brandon M</creator><creator>Vandewiele, Gilles</creator><creator>Narayanan, Arvind</creator><general>Cornell University Library, arXiv.org</general><scope>8FE</scope><scope>8FG</scope><scope>ABJCF</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>HCIFZ</scope><scope>L6V</scope><scope>M7S</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PTHSS</scope></search><sort><creationdate>20230919</creationdate><title>REFORMS: Reporting Standards for Machine Learning Based Science</title><author>Kapoor, Sayash ; Cantrell, Emily ; Kenny, Peng ; Pham, Thanh Hien ; Bail, Christopher A ; Gundersen, Odd Erik ; Hofman, Jake M ; Hullman, Jessica ; Lones, Michael A ; Malik, Momin M ; Nanayakkara, Priyanka ; Poldrack, Russell A ; Inioluwa, Deborah Raji ; Roberts, Michael ; Salganik, Matthew J ; Serra-Garcia, Marta ; Stewart, Brandon M ; Vandewiele, Gilles ; Narayanan, Arvind</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-proquest_journals_28514795873</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2023</creationdate><topic>Literature reviews</topic><topic>Machine learning</topic><topic>Reproducibility</topic><topic>Science</topic><toplevel>online_resources</toplevel><creatorcontrib>Kapoor, Sayash</creatorcontrib><creatorcontrib>Cantrell, Emily</creatorcontrib><creatorcontrib>Kenny, Peng</creatorcontrib><creatorcontrib>Pham, Thanh Hien</creatorcontrib><creatorcontrib>Bail, Christopher A</creatorcontrib><creatorcontrib>Gundersen, Odd Erik</creatorcontrib><creatorcontrib>Hofman, Jake M</creatorcontrib><creatorcontrib>Hullman, Jessica</creatorcontrib><creatorcontrib>Lones, Michael A</creatorcontrib><creatorcontrib>Malik, Momin M</creatorcontrib><creatorcontrib>Nanayakkara, Priyanka</creatorcontrib><creatorcontrib>Poldrack, Russell A</creatorcontrib><creatorcontrib>Inioluwa, Deborah Raji</creatorcontrib><creatorcontrib>Roberts, Michael</creatorcontrib><creatorcontrib>Salganik, Matthew J</creatorcontrib><creatorcontrib>Serra-Garcia, Marta</creatorcontrib><creatorcontrib>Stewart, Brandon M</creatorcontrib><creatorcontrib>Vandewiele, Gilles</creatorcontrib><creatorcontrib>Narayanan, Arvind</creatorcontrib><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>Materials Science & Engineering Collection</collection><collection>ProQuest Central (Alumni Edition)</collection><collection>ProQuest Central UK/Ireland</collection><collection>ProQuest Central Essentials</collection><collection>ProQuest Central</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central Korea</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Engineering Collection</collection><collection>Engineering Database</collection><collection>Publicly Available Content Database</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>Engineering Collection</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Kapoor, Sayash</au><au>Cantrell, Emily</au><au>Kenny, Peng</au><au>Pham, Thanh Hien</au><au>Bail, Christopher A</au><au>Gundersen, Odd Erik</au><au>Hofman, Jake M</au><au>Hullman, Jessica</au><au>Lones, Michael A</au><au>Malik, Momin M</au><au>Nanayakkara, Priyanka</au><au>Poldrack, Russell A</au><au>Inioluwa, Deborah Raji</au><au>Roberts, Michael</au><au>Salganik, Matthew J</au><au>Serra-Garcia, Marta</au><au>Stewart, Brandon M</au><au>Vandewiele, Gilles</au><au>Narayanan, Arvind</au><format>book</format><genre>document</genre><ristype>GEN</ristype><atitle>REFORMS: Reporting Standards for Machine Learning Based Science</atitle><jtitle>arXiv.org</jtitle><date>2023-09-19</date><risdate>2023</risdate><eissn>2331-8422</eissn><abstract>Machine learning (ML) methods are proliferating in scientific research. However, the adoption of these methods has been accompanied by failures of validity, reproducibility, and generalizability. These failures can hinder scientific progress, lead to false consensus around invalid claims, and undermine the credibility of ML-based science. ML methods are often applied and fail in similar ways across disciplines. Motivated by this observation, our goal is to provide clear reporting standards for ML-based science. Drawing from an extensive review of past literature, we present the REFORMS checklist (\(\textbf{Re}\)porting Standards \(\textbf{For}\) \(\textbf{M}\)achine Learning Based \(\textbf{S}\)cience). It consists of 32 questions and a paired set of guidelines. REFORMS was developed based on a consensus of 19 researchers across computer science, data science, mathematics, social sciences, and biomedical sciences. REFORMS can serve as a resource for researchers when designing and implementing a study, for referees when reviewing papers, and for journals when enforcing standards for transparency and reproducibility.</abstract><cop>Ithaca</cop><pub>Cornell University Library, arXiv.org</pub><oa>free_for_read</oa></addata></record> |
fulltext | fulltext |
identifier | EISSN: 2331-8422 |
ispartof | arXiv.org, 2023-09 |
issn | 2331-8422 |
language | eng |
recordid | cdi_proquest_journals_2851479587 |
source | Free E- Journals |
subjects | Literature reviews Machine learning Reproducibility Science |
title | REFORMS: Reporting Standards for Machine Learning Based Science |
url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-24T18%3A15%3A15IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest&rft_val_fmt=info:ofi/fmt:kev:mtx:book&rft.genre=document&rft.atitle=REFORMS:%20Reporting%20Standards%20for%20Machine%20Learning%20Based%20Science&rft.jtitle=arXiv.org&rft.au=Kapoor,%20Sayash&rft.date=2023-09-19&rft.eissn=2331-8422&rft_id=info:doi/&rft_dat=%3Cproquest%3E2851479587%3C/proquest%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2851479587&rft_id=info:pmid/&rfr_iscdi=true |