Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research
Systematic reviews and meta-analyses are crucial for advancing research, yet they are time-consuming and resource-demanding. Although machine learning and natural language processing algorithms may reduce this time and these resources, their performance has not been tested in education and education...
Gespeichert in:
Veröffentlicht in: | Educational psychology review 2024-03, Vol.36 (1), p.19, Article 19 |
---|---|
Hauptverfasser: | , , , , , , , , |
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
container_end_page | |
---|---|
container_issue | 1 |
container_start_page | 19 |
container_title | Educational psychology review |
container_volume | 36 |
creator | Campos, Diego G. Fütterer, Tim Gfrörer, Thomas Lavelle-Hill, Rosa Murayama, Kou König, Lars Hecht, Martin Zitzmann, Steffen Scherer, Ronny |
description | Systematic reviews and meta-analyses are crucial for advancing research, yet they are time-consuming and resource-demanding. Although machine learning and natural language processing algorithms may reduce this time and these resources, their performance has not been tested in education and educational psychology, and there is a lack of clear information on when researchers should stop the reviewing process. In this study, we conducted a retrospective screening simulation using 27 systematic reviews in education and educational psychology. We evaluated the sensitivity, specificity, and estimated time savings of several learning algorithms and heuristic stopping criteria. The results showed, on average, a 58% (
SD
= 19%) reduction in the screening workload of irrelevant records when using learning algorithms for abstract screening and an estimated time savings of 1.66 days (
SD
= 1.80). The learning algorithm random forests with sentence bidirectional encoder representations from transformers outperformed other algorithms. This finding emphasizes the importance of incorporating semantic and contextual information during feature extraction and modeling in the screening process. Furthermore, we found that 95% of all relevant abstracts within a given dataset can be retrieved using heuristic stopping rules. Specifically, an approach that stops the screening process after classifying 20% of records and consecutively classifying 5% of irrelevant papers yielded the most significant gains in terms of specificity (
M
= 42%,
SD
= 28%). However, the performance of the heuristic stopping criteria depended on the learning algorithm used and the length and proportion of relevant papers in an abstract collection. Our study provides empirical evidence on the performance of machine learning screening algorithms for abstract screening in systematic reviews in education and educational psychology. |
doi_str_mv | 10.1007/s10648-024-09862-5 |
format | Article |
fullrecord | <record><control><sourceid>proquest_crist</sourceid><recordid>TN_cdi_cristin_nora_10852_108955</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2923591665</sourcerecordid><originalsourceid>FETCH-LOGICAL-c388t-5d93bb1c5c837d5f110ff3deae892e8f69a0aff1eda6a576ecbf02740330d803</originalsourceid><addsrcrecordid>eNp9kc9uEzEQxlcIJErhBThZ4srC2F7verlFUUsqBZBID71ZjnecuErsxXaK8lC8Y71d_ty42Nb4930zmq-q3lL4QAG6j4lC28gaWFNDL1tWi2fVBRUdrzlv756XNzRQd6y9e1m9SukeAPqu4RfVr42JiN75HdkcdcwY35OvIZOVjgPGT2RBluE46qize0Cy8PpwTi6RYMkXbfbOI1mjjrP-r9PisAvR5f0xEe0HssJTdCk7QzY5jONELMs3RqeJDZFszinjUU_Ad3xw-DMR58nVcDKlFkrLUk6li9m_rl5YfUj45vd9Wd1eX90uV_X62-eb5WJdGy5lrsXQ8-2WGmEk7wZhKQVr-YAaZc9Q2rbXoK2lOOhWi65Fs7XAugY4h0ECv6zIbGue5vbKh6gVBSnYdPZCFOTdjIwx_Dhhyuo-nGKZNSnWMy562rYTxf4YhZQiWjVGV9Z8LjZqCk7NwakSnHoKTk0iPotSgf0O4z_r_6geAcQsnd8</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2923591665</pqid></control><display><type>article</type><title>Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research</title><source>NORA - Norwegian Open Research Archives</source><source>Springer Nature - Complete Springer Journals</source><creator>Campos, Diego G. ; Fütterer, Tim ; Gfrörer, Thomas ; Lavelle-Hill, Rosa ; Murayama, Kou ; König, Lars ; Hecht, Martin ; Zitzmann, Steffen ; Scherer, Ronny</creator><creatorcontrib>Campos, Diego G. ; Fütterer, Tim ; Gfrörer, Thomas ; Lavelle-Hill, Rosa ; Murayama, Kou ; König, Lars ; Hecht, Martin ; Zitzmann, Steffen ; Scherer, Ronny</creatorcontrib><description>Systematic reviews and meta-analyses are crucial for advancing research, yet they are time-consuming and resource-demanding. Although machine learning and natural language processing algorithms may reduce this time and these resources, their performance has not been tested in education and educational psychology, and there is a lack of clear information on when researchers should stop the reviewing process. In this study, we conducted a retrospective screening simulation using 27 systematic reviews in education and educational psychology. We evaluated the sensitivity, specificity, and estimated time savings of several learning algorithms and heuristic stopping criteria. The results showed, on average, a 58% (
SD
= 19%) reduction in the screening workload of irrelevant records when using learning algorithms for abstract screening and an estimated time savings of 1.66 days (
SD
= 1.80). The learning algorithm random forests with sentence bidirectional encoder representations from transformers outperformed other algorithms. This finding emphasizes the importance of incorporating semantic and contextual information during feature extraction and modeling in the screening process. Furthermore, we found that 95% of all relevant abstracts within a given dataset can be retrieved using heuristic stopping rules. Specifically, an approach that stops the screening process after classifying 20% of records and consecutively classifying 5% of irrelevant papers yielded the most significant gains in terms of specificity (
M
= 42%,
SD
= 28%). However, the performance of the heuristic stopping criteria depended on the learning algorithm used and the length and proportion of relevant papers in an abstract collection. Our study provides empirical evidence on the performance of machine learning screening algorithms for abstract screening in systematic reviews in education and educational psychology.</description><identifier>ISSN: 1040-726X</identifier><identifier>EISSN: 1573-336X</identifier><identifier>DOI: 10.1007/s10648-024-09862-5</identifier><language>eng</language><publisher>New York: Springer US</publisher><subject>Algorithms ; Artificial Intelligence ; Child and School Psychology ; Comparative Analysis ; Education ; Educational Psychology ; Educational Research ; Heuristic ; Heuristics ; Language Processing ; Learning and Instruction ; Machine learning ; Mathematics ; Meta Analysis ; Resistance (Psychology) ; Review Article ; Semantics ; Systematic review</subject><ispartof>Educational psychology review, 2024-03, Vol.36 (1), p.19, Article 19</ispartof><rights>The Author(s) 2024</rights><rights>The Author(s) 2024. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><rights>info:eu-repo/semantics/openAccess</rights><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c388t-5d93bb1c5c837d5f110ff3deae892e8f69a0aff1eda6a576ecbf02740330d803</citedby><cites>FETCH-LOGICAL-c388t-5d93bb1c5c837d5f110ff3deae892e8f69a0aff1eda6a576ecbf02740330d803</cites><orcidid>0000-0001-5399-9557 ; 0000-0002-5168-4911 ; 0000-0002-4817-2296 ; 0009-0000-3525-771X ; 0000-0003-3630-0710 ; 0000-0002-8820-5881 ; 0000-0003-2902-9600 ; 0000-0002-1767-9828 ; 0000-0002-7595-4736</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktopdf>$$Uhttps://link.springer.com/content/pdf/10.1007/s10648-024-09862-5$$EPDF$$P50$$Gspringer$$Hfree_for_read</linktopdf><linktohtml>$$Uhttps://link.springer.com/10.1007/s10648-024-09862-5$$EHTML$$P50$$Gspringer$$Hfree_for_read</linktohtml><link.rule.ids>230,314,776,780,881,26546,27903,27904,41467,42536,51298</link.rule.ids></links><search><creatorcontrib>Campos, Diego G.</creatorcontrib><creatorcontrib>Fütterer, Tim</creatorcontrib><creatorcontrib>Gfrörer, Thomas</creatorcontrib><creatorcontrib>Lavelle-Hill, Rosa</creatorcontrib><creatorcontrib>Murayama, Kou</creatorcontrib><creatorcontrib>König, Lars</creatorcontrib><creatorcontrib>Hecht, Martin</creatorcontrib><creatorcontrib>Zitzmann, Steffen</creatorcontrib><creatorcontrib>Scherer, Ronny</creatorcontrib><title>Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research</title><title>Educational psychology review</title><addtitle>Educ Psychol Rev</addtitle><description>Systematic reviews and meta-analyses are crucial for advancing research, yet they are time-consuming and resource-demanding. Although machine learning and natural language processing algorithms may reduce this time and these resources, their performance has not been tested in education and educational psychology, and there is a lack of clear information on when researchers should stop the reviewing process. In this study, we conducted a retrospective screening simulation using 27 systematic reviews in education and educational psychology. We evaluated the sensitivity, specificity, and estimated time savings of several learning algorithms and heuristic stopping criteria. The results showed, on average, a 58% (
SD
= 19%) reduction in the screening workload of irrelevant records when using learning algorithms for abstract screening and an estimated time savings of 1.66 days (
SD
= 1.80). The learning algorithm random forests with sentence bidirectional encoder representations from transformers outperformed other algorithms. This finding emphasizes the importance of incorporating semantic and contextual information during feature extraction and modeling in the screening process. Furthermore, we found that 95% of all relevant abstracts within a given dataset can be retrieved using heuristic stopping rules. Specifically, an approach that stops the screening process after classifying 20% of records and consecutively classifying 5% of irrelevant papers yielded the most significant gains in terms of specificity (
M
= 42%,
SD
= 28%). However, the performance of the heuristic stopping criteria depended on the learning algorithm used and the length and proportion of relevant papers in an abstract collection. Our study provides empirical evidence on the performance of machine learning screening algorithms for abstract screening in systematic reviews in education and educational psychology.</description><subject>Algorithms</subject><subject>Artificial Intelligence</subject><subject>Child and School Psychology</subject><subject>Comparative Analysis</subject><subject>Education</subject><subject>Educational Psychology</subject><subject>Educational Research</subject><subject>Heuristic</subject><subject>Heuristics</subject><subject>Language Processing</subject><subject>Learning and Instruction</subject><subject>Machine learning</subject><subject>Mathematics</subject><subject>Meta Analysis</subject><subject>Resistance (Psychology)</subject><subject>Review Article</subject><subject>Semantics</subject><subject>Systematic review</subject><issn>1040-726X</issn><issn>1573-336X</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><sourceid>C6C</sourceid><sourceid>3HK</sourceid><recordid>eNp9kc9uEzEQxlcIJErhBThZ4srC2F7verlFUUsqBZBID71ZjnecuErsxXaK8lC8Y71d_ty42Nb4930zmq-q3lL4QAG6j4lC28gaWFNDL1tWi2fVBRUdrzlv756XNzRQd6y9e1m9SukeAPqu4RfVr42JiN75HdkcdcwY35OvIZOVjgPGT2RBluE46qize0Cy8PpwTi6RYMkXbfbOI1mjjrP-r9PisAvR5f0xEe0HssJTdCk7QzY5jONELMs3RqeJDZFszinjUU_Ad3xw-DMR58nVcDKlFkrLUk6li9m_rl5YfUj45vd9Wd1eX90uV_X62-eb5WJdGy5lrsXQ8-2WGmEk7wZhKQVr-YAaZc9Q2rbXoK2lOOhWi65Fs7XAugY4h0ECv6zIbGue5vbKh6gVBSnYdPZCFOTdjIwx_Dhhyuo-nGKZNSnWMy562rYTxf4YhZQiWjVGV9Z8LjZqCk7NwakSnHoKTk0iPotSgf0O4z_r_6geAcQsnd8</recordid><startdate>20240301</startdate><enddate>20240301</enddate><creator>Campos, Diego G.</creator><creator>Fütterer, Tim</creator><creator>Gfrörer, Thomas</creator><creator>Lavelle-Hill, Rosa</creator><creator>Murayama, Kou</creator><creator>König, Lars</creator><creator>Hecht, Martin</creator><creator>Zitzmann, Steffen</creator><creator>Scherer, Ronny</creator><general>Springer US</general><general>Springer Nature B.V</general><scope>C6C</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>3HK</scope><orcidid>https://orcid.org/0000-0001-5399-9557</orcidid><orcidid>https://orcid.org/0000-0002-5168-4911</orcidid><orcidid>https://orcid.org/0000-0002-4817-2296</orcidid><orcidid>https://orcid.org/0009-0000-3525-771X</orcidid><orcidid>https://orcid.org/0000-0003-3630-0710</orcidid><orcidid>https://orcid.org/0000-0002-8820-5881</orcidid><orcidid>https://orcid.org/0000-0003-2902-9600</orcidid><orcidid>https://orcid.org/0000-0002-1767-9828</orcidid><orcidid>https://orcid.org/0000-0002-7595-4736</orcidid></search><sort><creationdate>20240301</creationdate><title>Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research</title><author>Campos, Diego G. ; Fütterer, Tim ; Gfrörer, Thomas ; Lavelle-Hill, Rosa ; Murayama, Kou ; König, Lars ; Hecht, Martin ; Zitzmann, Steffen ; Scherer, Ronny</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c388t-5d93bb1c5c837d5f110ff3deae892e8f69a0aff1eda6a576ecbf02740330d803</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><topic>Algorithms</topic><topic>Artificial Intelligence</topic><topic>Child and School Psychology</topic><topic>Comparative Analysis</topic><topic>Education</topic><topic>Educational Psychology</topic><topic>Educational Research</topic><topic>Heuristic</topic><topic>Heuristics</topic><topic>Language Processing</topic><topic>Learning and Instruction</topic><topic>Machine learning</topic><topic>Mathematics</topic><topic>Meta Analysis</topic><topic>Resistance (Psychology)</topic><topic>Review Article</topic><topic>Semantics</topic><topic>Systematic review</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Campos, Diego G.</creatorcontrib><creatorcontrib>Fütterer, Tim</creatorcontrib><creatorcontrib>Gfrörer, Thomas</creatorcontrib><creatorcontrib>Lavelle-Hill, Rosa</creatorcontrib><creatorcontrib>Murayama, Kou</creatorcontrib><creatorcontrib>König, Lars</creatorcontrib><creatorcontrib>Hecht, Martin</creatorcontrib><creatorcontrib>Zitzmann, Steffen</creatorcontrib><creatorcontrib>Scherer, Ronny</creatorcontrib><collection>Springer Nature OA Free Journals</collection><collection>CrossRef</collection><collection>NORA - Norwegian Open Research Archives</collection><jtitle>Educational psychology review</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Campos, Diego G.</au><au>Fütterer, Tim</au><au>Gfrörer, Thomas</au><au>Lavelle-Hill, Rosa</au><au>Murayama, Kou</au><au>König, Lars</au><au>Hecht, Martin</au><au>Zitzmann, Steffen</au><au>Scherer, Ronny</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research</atitle><jtitle>Educational psychology review</jtitle><stitle>Educ Psychol Rev</stitle><date>2024-03-01</date><risdate>2024</risdate><volume>36</volume><issue>1</issue><spage>19</spage><pages>19-</pages><artnum>19</artnum><issn>1040-726X</issn><eissn>1573-336X</eissn><abstract>Systematic reviews and meta-analyses are crucial for advancing research, yet they are time-consuming and resource-demanding. Although machine learning and natural language processing algorithms may reduce this time and these resources, their performance has not been tested in education and educational psychology, and there is a lack of clear information on when researchers should stop the reviewing process. In this study, we conducted a retrospective screening simulation using 27 systematic reviews in education and educational psychology. We evaluated the sensitivity, specificity, and estimated time savings of several learning algorithms and heuristic stopping criteria. The results showed, on average, a 58% (
SD
= 19%) reduction in the screening workload of irrelevant records when using learning algorithms for abstract screening and an estimated time savings of 1.66 days (
SD
= 1.80). The learning algorithm random forests with sentence bidirectional encoder representations from transformers outperformed other algorithms. This finding emphasizes the importance of incorporating semantic and contextual information during feature extraction and modeling in the screening process. Furthermore, we found that 95% of all relevant abstracts within a given dataset can be retrieved using heuristic stopping rules. Specifically, an approach that stops the screening process after classifying 20% of records and consecutively classifying 5% of irrelevant papers yielded the most significant gains in terms of specificity (
M
= 42%,
SD
= 28%). However, the performance of the heuristic stopping criteria depended on the learning algorithm used and the length and proportion of relevant papers in an abstract collection. Our study provides empirical evidence on the performance of machine learning screening algorithms for abstract screening in systematic reviews in education and educational psychology.</abstract><cop>New York</cop><pub>Springer US</pub><doi>10.1007/s10648-024-09862-5</doi><orcidid>https://orcid.org/0000-0001-5399-9557</orcidid><orcidid>https://orcid.org/0000-0002-5168-4911</orcidid><orcidid>https://orcid.org/0000-0002-4817-2296</orcidid><orcidid>https://orcid.org/0009-0000-3525-771X</orcidid><orcidid>https://orcid.org/0000-0003-3630-0710</orcidid><orcidid>https://orcid.org/0000-0002-8820-5881</orcidid><orcidid>https://orcid.org/0000-0003-2902-9600</orcidid><orcidid>https://orcid.org/0000-0002-1767-9828</orcidid><orcidid>https://orcid.org/0000-0002-7595-4736</orcidid><oa>free_for_read</oa></addata></record> |
fulltext | fulltext |
identifier | ISSN: 1040-726X |
ispartof | Educational psychology review, 2024-03, Vol.36 (1), p.19, Article 19 |
issn | 1040-726X 1573-336X |
language | eng |
recordid | cdi_cristin_nora_10852_108955 |
source | NORA - Norwegian Open Research Archives; Springer Nature - Complete Springer Journals |
subjects | Algorithms Artificial Intelligence Child and School Psychology Comparative Analysis Education Educational Psychology Educational Research Heuristic Heuristics Language Processing Learning and Instruction Machine learning Mathematics Meta Analysis Resistance (Psychology) Review Article Semantics Systematic review |
title | Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research |
url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-23T02%3A33%3A28IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_crist&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Screening%20Smarter,%20Not%20Harder:%20A%20Comparative%20Analysis%20of%20Machine%20Learning%20Screening%20Algorithms%20and%20Heuristic%20Stopping%20Criteria%20for%20Systematic%20Reviews%20in%20Educational%20Research&rft.jtitle=Educational%20psychology%20review&rft.au=Campos,%20Diego%20G.&rft.date=2024-03-01&rft.volume=36&rft.issue=1&rft.spage=19&rft.pages=19-&rft.artnum=19&rft.issn=1040-726X&rft.eissn=1573-336X&rft_id=info:doi/10.1007/s10648-024-09862-5&rft_dat=%3Cproquest_crist%3E2923591665%3C/proquest_crist%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2923591665&rft_id=info:pmid/&rfr_iscdi=true |