Evaluating the validity of the nursing statements algorithmically generated based on the International Classifications of Nursing Practice for respiratory nursing care using large language models

This study aims to facilitate the creation of quality standardized nursing statements in South Korea's hospitals using algorithmic generation based on the International Classifications of Nursing Practice (ICNP) and evaluation through Large Language Models. We algorithmically generated 15 972 s...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Journal of the American Medical Informatics Association : JAMIA 2024-05, Vol.31 (6), p.1397-1403
Hauptverfasser: Kim, Hyeoneui, Park, Hyewon, Kang, Sunghoon, Kim, Jinsol, Kim, Jeongha, Jung, Jinsun, Taira, Ricky
Format: Artikel
Sprache:eng
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page 1403
container_issue 6
container_start_page 1397
container_title Journal of the American Medical Informatics Association : JAMIA
container_volume 31
creator Kim, Hyeoneui
Park, Hyewon
Kang, Sunghoon
Kim, Jinsol
Kim, Jeongha
Jung, Jinsun
Taira, Ricky
description This study aims to facilitate the creation of quality standardized nursing statements in South Korea's hospitals using algorithmic generation based on the International Classifications of Nursing Practice (ICNP) and evaluation through Large Language Models. We algorithmically generated 15 972 statements related to acute respiratory care using 117 concepts and concept composition models of ICNP. Human reviewers, Generative Pre-trained Transformers 4.0 (GPT-4.0), and Bio_Clinical Bidirectional Encoder Representations from Transformers (BERT) evaluated the generated statements for validity. The evaluation by GPT-4.0 and Bio_ClinicalBERT was conducted with and without contextual information and training. Of the generated statements, 2207 were deemed valid by expert reviewers. GPT-4.0 showed a zero-shot  AUC of 0.857, which aggravated with contextual information. Bio_ClinicalBERT, after training, significantly improved, reaching an AUC of 0.998. Bio_ClinicalBERT effectively validates auto-generated nursing statements, offering a promising solution to enhance and streamline healthcare documentation processes.
doi_str_mv 10.1093/jamia/ocae070
format Article
fullrecord <record><control><sourceid>proquest_cross</sourceid><recordid>TN_cdi_proquest_miscellaneous_3041232148</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>3041232148</sourcerecordid><originalsourceid>FETCH-LOGICAL-c249t-b7e0e175538a1d6352880b554b442c3c45a43d2c0714e4404791f294cd0010843</originalsourceid><addsrcrecordid>eNo9kU1v1DAQhi0EoqVw5Ip85BLqzzg5olWBShVwAIlbNHEmqSvHXmwHaX8ffwzvdullPON55xnLLyFvOfvAWS-vH2B1cB0tIDPsGbnkWpimN-rX85qz1jSaCXNBXuX8wBhvhdQvyYXsWsl0116Svzd_wG9QXFhouUdaKze5cqBxPtVhS_nYywUKrhhKpuCXmFy5X50F7w90wYCpdic6Qq4xhtPkbSiYQiXHAJ7uPOTs5jpyvMhH_Ncz-nsCW5xFOsdEE-a9q7SYDk-7LSSk2yn1kBasMSwb1GSNE_r8mryYwWd8cz6vyM9PNz92X5q7b59vdx_vGitUX5rRIENutJYd8KmVWnQdG7VWo1LCSqs0KDkJywxXqBRTpuez6JWd6sexTskr8v6Ru0_x94a5DKvLFn19DsYtD5IpLqTgqqvS5lFqU8w54Tzsk1shHQbOhqNvw8m34exb1b87o7dxxelJ_d8o-Q-Cw5qV</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>3041232148</pqid></control><display><type>article</type><title>Evaluating the validity of the nursing statements algorithmically generated based on the International Classifications of Nursing Practice for respiratory nursing care using large language models</title><source>Oxford University Press Journals All Titles (1996-Current)</source><creator>Kim, Hyeoneui ; Park, Hyewon ; Kang, Sunghoon ; Kim, Jinsol ; Kim, Jeongha ; Jung, Jinsun ; Taira, Ricky</creator><creatorcontrib>Kim, Hyeoneui ; Park, Hyewon ; Kang, Sunghoon ; Kim, Jinsol ; Kim, Jeongha ; Jung, Jinsun ; Taira, Ricky</creatorcontrib><description>This study aims to facilitate the creation of quality standardized nursing statements in South Korea's hospitals using algorithmic generation based on the International Classifications of Nursing Practice (ICNP) and evaluation through Large Language Models. We algorithmically generated 15 972 statements related to acute respiratory care using 117 concepts and concept composition models of ICNP. Human reviewers, Generative Pre-trained Transformers 4.0 (GPT-4.0), and Bio_Clinical Bidirectional Encoder Representations from Transformers (BERT) evaluated the generated statements for validity. The evaluation by GPT-4.0 and Bio_ClinicalBERT was conducted with and without contextual information and training. Of the generated statements, 2207 were deemed valid by expert reviewers. GPT-4.0 showed a zero-shot  AUC of 0.857, which aggravated with contextual information. Bio_ClinicalBERT, after training, significantly improved, reaching an AUC of 0.998. Bio_ClinicalBERT effectively validates auto-generated nursing statements, offering a promising solution to enhance and streamline healthcare documentation processes.</description><identifier>ISSN: 1067-5027</identifier><identifier>EISSN: 1527-974X</identifier><identifier>DOI: 10.1093/jamia/ocae070</identifier><identifier>PMID: 38630586</identifier><language>eng</language><publisher>England</publisher><ispartof>Journal of the American Medical Informatics Association : JAMIA, 2024-05, Vol.31 (6), p.1397-1403</ispartof><rights>The Author(s) 2024. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissions@oup.com.</rights><lds50>peer_reviewed</lds50><woscitedreferencessubscribed>false</woscitedreferencessubscribed><cites>FETCH-LOGICAL-c249t-b7e0e175538a1d6352880b554b442c3c45a43d2c0714e4404791f294cd0010843</cites><orcidid>0000-0002-9993-5062 ; 0000-0002-5931-7286</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>314,780,784,27924,27925</link.rule.ids><backlink>$$Uhttps://www.ncbi.nlm.nih.gov/pubmed/38630586$$D View this record in MEDLINE/PubMed$$Hfree_for_read</backlink></links><search><creatorcontrib>Kim, Hyeoneui</creatorcontrib><creatorcontrib>Park, Hyewon</creatorcontrib><creatorcontrib>Kang, Sunghoon</creatorcontrib><creatorcontrib>Kim, Jinsol</creatorcontrib><creatorcontrib>Kim, Jeongha</creatorcontrib><creatorcontrib>Jung, Jinsun</creatorcontrib><creatorcontrib>Taira, Ricky</creatorcontrib><title>Evaluating the validity of the nursing statements algorithmically generated based on the International Classifications of Nursing Practice for respiratory nursing care using large language models</title><title>Journal of the American Medical Informatics Association : JAMIA</title><addtitle>J Am Med Inform Assoc</addtitle><description>This study aims to facilitate the creation of quality standardized nursing statements in South Korea's hospitals using algorithmic generation based on the International Classifications of Nursing Practice (ICNP) and evaluation through Large Language Models. We algorithmically generated 15 972 statements related to acute respiratory care using 117 concepts and concept composition models of ICNP. Human reviewers, Generative Pre-trained Transformers 4.0 (GPT-4.0), and Bio_Clinical Bidirectional Encoder Representations from Transformers (BERT) evaluated the generated statements for validity. The evaluation by GPT-4.0 and Bio_ClinicalBERT was conducted with and without contextual information and training. Of the generated statements, 2207 were deemed valid by expert reviewers. GPT-4.0 showed a zero-shot  AUC of 0.857, which aggravated with contextual information. Bio_ClinicalBERT, after training, significantly improved, reaching an AUC of 0.998. Bio_ClinicalBERT effectively validates auto-generated nursing statements, offering a promising solution to enhance and streamline healthcare documentation processes.</description><issn>1067-5027</issn><issn>1527-974X</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><recordid>eNo9kU1v1DAQhi0EoqVw5Ip85BLqzzg5olWBShVwAIlbNHEmqSvHXmwHaX8ffwzvdullPON55xnLLyFvOfvAWS-vH2B1cB0tIDPsGbnkWpimN-rX85qz1jSaCXNBXuX8wBhvhdQvyYXsWsl0116Svzd_wG9QXFhouUdaKze5cqBxPtVhS_nYywUKrhhKpuCXmFy5X50F7w90wYCpdic6Qq4xhtPkbSiYQiXHAJ7uPOTs5jpyvMhH_Ncz-nsCW5xFOsdEE-a9q7SYDk-7LSSk2yn1kBasMSwb1GSNE_r8mryYwWd8cz6vyM9PNz92X5q7b59vdx_vGitUX5rRIENutJYd8KmVWnQdG7VWo1LCSqs0KDkJywxXqBRTpuez6JWd6sexTskr8v6Ru0_x94a5DKvLFn19DsYtD5IpLqTgqqvS5lFqU8w54Tzsk1shHQbOhqNvw8m34exb1b87o7dxxelJ_d8o-Q-Cw5qV</recordid><startdate>20240520</startdate><enddate>20240520</enddate><creator>Kim, Hyeoneui</creator><creator>Park, Hyewon</creator><creator>Kang, Sunghoon</creator><creator>Kim, Jinsol</creator><creator>Kim, Jeongha</creator><creator>Jung, Jinsun</creator><creator>Taira, Ricky</creator><scope>NPM</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>7X8</scope><orcidid>https://orcid.org/0000-0002-9993-5062</orcidid><orcidid>https://orcid.org/0000-0002-5931-7286</orcidid></search><sort><creationdate>20240520</creationdate><title>Evaluating the validity of the nursing statements algorithmically generated based on the International Classifications of Nursing Practice for respiratory nursing care using large language models</title><author>Kim, Hyeoneui ; Park, Hyewon ; Kang, Sunghoon ; Kim, Jinsol ; Kim, Jeongha ; Jung, Jinsun ; Taira, Ricky</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c249t-b7e0e175538a1d6352880b554b442c3c45a43d2c0714e4404791f294cd0010843</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Kim, Hyeoneui</creatorcontrib><creatorcontrib>Park, Hyewon</creatorcontrib><creatorcontrib>Kang, Sunghoon</creatorcontrib><creatorcontrib>Kim, Jinsol</creatorcontrib><creatorcontrib>Kim, Jeongha</creatorcontrib><creatorcontrib>Jung, Jinsun</creatorcontrib><creatorcontrib>Taira, Ricky</creatorcontrib><collection>PubMed</collection><collection>CrossRef</collection><collection>MEDLINE - Academic</collection><jtitle>Journal of the American Medical Informatics Association : JAMIA</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Kim, Hyeoneui</au><au>Park, Hyewon</au><au>Kang, Sunghoon</au><au>Kim, Jinsol</au><au>Kim, Jeongha</au><au>Jung, Jinsun</au><au>Taira, Ricky</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Evaluating the validity of the nursing statements algorithmically generated based on the International Classifications of Nursing Practice for respiratory nursing care using large language models</atitle><jtitle>Journal of the American Medical Informatics Association : JAMIA</jtitle><addtitle>J Am Med Inform Assoc</addtitle><date>2024-05-20</date><risdate>2024</risdate><volume>31</volume><issue>6</issue><spage>1397</spage><epage>1403</epage><pages>1397-1403</pages><issn>1067-5027</issn><eissn>1527-974X</eissn><abstract>This study aims to facilitate the creation of quality standardized nursing statements in South Korea's hospitals using algorithmic generation based on the International Classifications of Nursing Practice (ICNP) and evaluation through Large Language Models. We algorithmically generated 15 972 statements related to acute respiratory care using 117 concepts and concept composition models of ICNP. Human reviewers, Generative Pre-trained Transformers 4.0 (GPT-4.0), and Bio_Clinical Bidirectional Encoder Representations from Transformers (BERT) evaluated the generated statements for validity. The evaluation by GPT-4.0 and Bio_ClinicalBERT was conducted with and without contextual information and training. Of the generated statements, 2207 were deemed valid by expert reviewers. GPT-4.0 showed a zero-shot  AUC of 0.857, which aggravated with contextual information. Bio_ClinicalBERT, after training, significantly improved, reaching an AUC of 0.998. Bio_ClinicalBERT effectively validates auto-generated nursing statements, offering a promising solution to enhance and streamline healthcare documentation processes.</abstract><cop>England</cop><pmid>38630586</pmid><doi>10.1093/jamia/ocae070</doi><tpages>7</tpages><orcidid>https://orcid.org/0000-0002-9993-5062</orcidid><orcidid>https://orcid.org/0000-0002-5931-7286</orcidid></addata></record>
fulltext fulltext
identifier ISSN: 1067-5027
ispartof Journal of the American Medical Informatics Association : JAMIA, 2024-05, Vol.31 (6), p.1397-1403
issn 1067-5027
1527-974X
language eng
recordid cdi_proquest_miscellaneous_3041232148
source Oxford University Press Journals All Titles (1996-Current)
title Evaluating the validity of the nursing statements algorithmically generated based on the International Classifications of Nursing Practice for respiratory nursing care using large language models
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-24T21%3A34%3A09IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_cross&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Evaluating%20the%20validity%20of%20the%20nursing%20statements%20algorithmically%20generated%20based%20on%20the%20International%20Classifications%20of%20Nursing%20Practice%20for%20respiratory%20nursing%20care%20using%20large%20language%20models&rft.jtitle=Journal%20of%20the%20American%20Medical%20Informatics%20Association%20:%20JAMIA&rft.au=Kim,%20Hyeoneui&rft.date=2024-05-20&rft.volume=31&rft.issue=6&rft.spage=1397&rft.epage=1403&rft.pages=1397-1403&rft.issn=1067-5027&rft.eissn=1527-974X&rft_id=info:doi/10.1093/jamia/ocae070&rft_dat=%3Cproquest_cross%3E3041232148%3C/proquest_cross%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=3041232148&rft_id=info:pmid/38630586&rfr_iscdi=true