Evaluation of ChatGPT-Generated Educational Patient Pamphlets for Common Interventional Radiology Procedures
This study aimed to evaluate the accuracy and reliability of educational patient pamphlets created by ChatGPT, a large language model, for common interventional radiology (IR) procedures. Twenty frequently performed IR procedures were selected, and five users were tasked to independently request Cha...
Gespeichert in:
| Veröffentlicht in: | Academic radiology 2024-11, Vol.31 (11), p.4548-4553 |
|---|---|
| Hauptverfasser: | , , , , , , , , , |
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | 4553 |
|---|---|
| container_issue | 11 |
| container_start_page | 4548 |
| container_title | Academic radiology |
| container_volume | 31 |
| creator | Kooraki, Soheil Hosseiny, Melina Jalili, Mohamamd H. Rahsepar, Amir Ali Imanzadeh, Amir Kim, Grace Hyun Hassani, Cameron Abtin, Fereidoun Moriarty, John M. Bedayat, Arash |
| description | This study aimed to evaluate the accuracy and reliability of educational patient pamphlets created by ChatGPT, a large language model, for common interventional radiology (IR) procedures.
Twenty frequently performed IR procedures were selected, and five users were tasked to independently request ChatGPT to generate educational patient pamphlets for each procedure using identical commands. Subsequently, two independent radiologists assessed the content, quality, and accuracy of the pamphlets. The review focused on identifying potential errors, inaccuracies, the consistency of pamphlets.
In a thorough analysis of the education pamphlets, we identified shortcomings in 30% (30/100) of pamphlets, with a total of 34 specific inaccuracies, including missing information about sedation for the procedure (10/34), inaccuracies related to specific procedural-related complications (8/34). A key-word co-occurrence network showed consistent themes within each group of pamphlets, while a line-by-line comparison at the level of users and across different procedures showed statistically significant inconsistencies (P |
| doi_str_mv | 10.1016/j.acra.2024.05.024 |
| format | Article |
| fullrecord | <record><control><sourceid>proquest_cross</sourceid><recordid>TN_cdi_proquest_miscellaneous_3065276698</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><els_id>S1076633224003076</els_id><sourcerecordid>3065276698</sourcerecordid><originalsourceid>FETCH-LOGICAL-c237t-68d7d65294b320edb1cbd1f7c27b784878ddb031e47a5ec73ed99d987054a54f3</originalsourceid><addsrcrecordid>eNp9kNFq2zAYhcXoaNK0L9CL4sve2JMsWZJhNyNkWaGwMNJrIUu_FwfbyiQ50Lfps_TJpizZLnd1DvznHPg_hO4JLggm_NO-0MbrosQlK3BVJPmA5kQKmTPM-FXyWPCcU1rO0E0Ie4xJxSW9RjMqJa1ZJedoWB11P-nYuTFzbbbc6bjebPM1jOB1BJut7GT-nHWfbZKBMSYdDrseYsha57OlG4bUfhoj-GM6n7M_tO1c736-vr9tvDNgJw_hFn1sdR_g7qIL9PJ1tV1-y5-_r5-WX55zU1IRcy6tsLwqa9bQEoNtiGksaYUpRSMkSx9a22BKgAldgREUbF3bWgpcMV2xli7Q43n34N2vCUJUQxcM9L0ewU1BUZzWBee1TNHyHDXeheChVQffDdq_KoLVCbPaqxNmdcKscKWSpNLDZX9qBrD_Kn-5psDncwDSl8cOvAomoUsYOg8mKuu6_-3_BhEPkKc</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>3065276698</pqid></control><display><type>article</type><title>Evaluation of ChatGPT-Generated Educational Patient Pamphlets for Common Interventional Radiology Procedures</title><source>ScienceDirect Journals (5 years ago - present)</source><creator>Kooraki, Soheil ; Hosseiny, Melina ; Jalili, Mohamamd H. ; Rahsepar, Amir Ali ; Imanzadeh, Amir ; Kim, Grace Hyun ; Hassani, Cameron ; Abtin, Fereidoun ; Moriarty, John M. ; Bedayat, Arash</creator><creatorcontrib>Kooraki, Soheil ; Hosseiny, Melina ; Jalili, Mohamamd H. ; Rahsepar, Amir Ali ; Imanzadeh, Amir ; Kim, Grace Hyun ; Hassani, Cameron ; Abtin, Fereidoun ; Moriarty, John M. ; Bedayat, Arash</creatorcontrib><description>This study aimed to evaluate the accuracy and reliability of educational patient pamphlets created by ChatGPT, a large language model, for common interventional radiology (IR) procedures.
Twenty frequently performed IR procedures were selected, and five users were tasked to independently request ChatGPT to generate educational patient pamphlets for each procedure using identical commands. Subsequently, two independent radiologists assessed the content, quality, and accuracy of the pamphlets. The review focused on identifying potential errors, inaccuracies, the consistency of pamphlets.
In a thorough analysis of the education pamphlets, we identified shortcomings in 30% (30/100) of pamphlets, with a total of 34 specific inaccuracies, including missing information about sedation for the procedure (10/34), inaccuracies related to specific procedural-related complications (8/34). A key-word co-occurrence network showed consistent themes within each group of pamphlets, while a line-by-line comparison at the level of users and across different procedures showed statistically significant inconsistencies (P < 0.001).
ChatGPT-generated education pamphlets demonstrated potential clinical relevance and fairly consistent terminology; however, the pamphlets were not entirely accurate and exhibited some shortcomings and inter-user structural variabilities. To ensure patient safety, future improvements and refinements in large language models are warranted, while maintaining human supervision and expert validation.</description><identifier>ISSN: 1076-6332</identifier><identifier>ISSN: 1878-4046</identifier><identifier>EISSN: 1878-4046</identifier><identifier>DOI: 10.1016/j.acra.2024.05.024</identifier><identifier>PMID: 38839458</identifier><language>eng</language><publisher>United States: Elsevier Inc</publisher><subject>Chat GPT ; Co-occurrence network graph ; Education ; Interventional radiology ; Large language models</subject><ispartof>Academic radiology, 2024-11, Vol.31 (11), p.4548-4553</ispartof><rights>2024 The Association of University Radiologists</rights><rights>Copyright © 2024 The Association of University Radiologists. Published by Elsevier Inc. All rights reserved.</rights><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><cites>FETCH-LOGICAL-c237t-68d7d65294b320edb1cbd1f7c27b784878ddb031e47a5ec73ed99d987054a54f3</cites><orcidid>0000-0001-8769-0224</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://dx.doi.org/10.1016/j.acra.2024.05.024$$EHTML$$P50$$Gelsevier$$Hfree_for_read</linktohtml><link.rule.ids>314,780,784,3550,27924,27925,45995</link.rule.ids><backlink>$$Uhttps://www.ncbi.nlm.nih.gov/pubmed/38839458$$D View this record in MEDLINE/PubMed$$Hfree_for_read</backlink></links><search><creatorcontrib>Kooraki, Soheil</creatorcontrib><creatorcontrib>Hosseiny, Melina</creatorcontrib><creatorcontrib>Jalili, Mohamamd H.</creatorcontrib><creatorcontrib>Rahsepar, Amir Ali</creatorcontrib><creatorcontrib>Imanzadeh, Amir</creatorcontrib><creatorcontrib>Kim, Grace Hyun</creatorcontrib><creatorcontrib>Hassani, Cameron</creatorcontrib><creatorcontrib>Abtin, Fereidoun</creatorcontrib><creatorcontrib>Moriarty, John M.</creatorcontrib><creatorcontrib>Bedayat, Arash</creatorcontrib><title>Evaluation of ChatGPT-Generated Educational Patient Pamphlets for Common Interventional Radiology Procedures</title><title>Academic radiology</title><addtitle>Acad Radiol</addtitle><description>This study aimed to evaluate the accuracy and reliability of educational patient pamphlets created by ChatGPT, a large language model, for common interventional radiology (IR) procedures.
Twenty frequently performed IR procedures were selected, and five users were tasked to independently request ChatGPT to generate educational patient pamphlets for each procedure using identical commands. Subsequently, two independent radiologists assessed the content, quality, and accuracy of the pamphlets. The review focused on identifying potential errors, inaccuracies, the consistency of pamphlets.
In a thorough analysis of the education pamphlets, we identified shortcomings in 30% (30/100) of pamphlets, with a total of 34 specific inaccuracies, including missing information about sedation for the procedure (10/34), inaccuracies related to specific procedural-related complications (8/34). A key-word co-occurrence network showed consistent themes within each group of pamphlets, while a line-by-line comparison at the level of users and across different procedures showed statistically significant inconsistencies (P < 0.001).
ChatGPT-generated education pamphlets demonstrated potential clinical relevance and fairly consistent terminology; however, the pamphlets were not entirely accurate and exhibited some shortcomings and inter-user structural variabilities. To ensure patient safety, future improvements and refinements in large language models are warranted, while maintaining human supervision and expert validation.</description><subject>Chat GPT</subject><subject>Co-occurrence network graph</subject><subject>Education</subject><subject>Interventional radiology</subject><subject>Large language models</subject><issn>1076-6332</issn><issn>1878-4046</issn><issn>1878-4046</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><recordid>eNp9kNFq2zAYhcXoaNK0L9CL4sve2JMsWZJhNyNkWaGwMNJrIUu_FwfbyiQ50Lfps_TJpizZLnd1DvznHPg_hO4JLggm_NO-0MbrosQlK3BVJPmA5kQKmTPM-FXyWPCcU1rO0E0Ie4xJxSW9RjMqJa1ZJedoWB11P-nYuTFzbbbc6bjebPM1jOB1BJut7GT-nHWfbZKBMSYdDrseYsha57OlG4bUfhoj-GM6n7M_tO1c736-vr9tvDNgJw_hFn1sdR_g7qIL9PJ1tV1-y5-_r5-WX55zU1IRcy6tsLwqa9bQEoNtiGksaYUpRSMkSx9a22BKgAldgREUbF3bWgpcMV2xli7Q43n34N2vCUJUQxcM9L0ewU1BUZzWBee1TNHyHDXeheChVQffDdq_KoLVCbPaqxNmdcKscKWSpNLDZX9qBrD_Kn-5psDncwDSl8cOvAomoUsYOg8mKuu6_-3_BhEPkKc</recordid><startdate>20241101</startdate><enddate>20241101</enddate><creator>Kooraki, Soheil</creator><creator>Hosseiny, Melina</creator><creator>Jalili, Mohamamd H.</creator><creator>Rahsepar, Amir Ali</creator><creator>Imanzadeh, Amir</creator><creator>Kim, Grace Hyun</creator><creator>Hassani, Cameron</creator><creator>Abtin, Fereidoun</creator><creator>Moriarty, John M.</creator><creator>Bedayat, Arash</creator><general>Elsevier Inc</general><scope>6I.</scope><scope>AAFTH</scope><scope>NPM</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>7X8</scope><orcidid>https://orcid.org/0000-0001-8769-0224</orcidid></search><sort><creationdate>20241101</creationdate><title>Evaluation of ChatGPT-Generated Educational Patient Pamphlets for Common Interventional Radiology Procedures</title><author>Kooraki, Soheil ; Hosseiny, Melina ; Jalili, Mohamamd H. ; Rahsepar, Amir Ali ; Imanzadeh, Amir ; Kim, Grace Hyun ; Hassani, Cameron ; Abtin, Fereidoun ; Moriarty, John M. ; Bedayat, Arash</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c237t-68d7d65294b320edb1cbd1f7c27b784878ddb031e47a5ec73ed99d987054a54f3</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><topic>Chat GPT</topic><topic>Co-occurrence network graph</topic><topic>Education</topic><topic>Interventional radiology</topic><topic>Large language models</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Kooraki, Soheil</creatorcontrib><creatorcontrib>Hosseiny, Melina</creatorcontrib><creatorcontrib>Jalili, Mohamamd H.</creatorcontrib><creatorcontrib>Rahsepar, Amir Ali</creatorcontrib><creatorcontrib>Imanzadeh, Amir</creatorcontrib><creatorcontrib>Kim, Grace Hyun</creatorcontrib><creatorcontrib>Hassani, Cameron</creatorcontrib><creatorcontrib>Abtin, Fereidoun</creatorcontrib><creatorcontrib>Moriarty, John M.</creatorcontrib><creatorcontrib>Bedayat, Arash</creatorcontrib><collection>ScienceDirect Open Access Titles</collection><collection>Elsevier:ScienceDirect:Open Access</collection><collection>PubMed</collection><collection>CrossRef</collection><collection>MEDLINE - Academic</collection><jtitle>Academic radiology</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Kooraki, Soheil</au><au>Hosseiny, Melina</au><au>Jalili, Mohamamd H.</au><au>Rahsepar, Amir Ali</au><au>Imanzadeh, Amir</au><au>Kim, Grace Hyun</au><au>Hassani, Cameron</au><au>Abtin, Fereidoun</au><au>Moriarty, John M.</au><au>Bedayat, Arash</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Evaluation of ChatGPT-Generated Educational Patient Pamphlets for Common Interventional Radiology Procedures</atitle><jtitle>Academic radiology</jtitle><addtitle>Acad Radiol</addtitle><date>2024-11-01</date><risdate>2024</risdate><volume>31</volume><issue>11</issue><spage>4548</spage><epage>4553</epage><pages>4548-4553</pages><issn>1076-6332</issn><issn>1878-4046</issn><eissn>1878-4046</eissn><abstract>This study aimed to evaluate the accuracy and reliability of educational patient pamphlets created by ChatGPT, a large language model, for common interventional radiology (IR) procedures.
Twenty frequently performed IR procedures were selected, and five users were tasked to independently request ChatGPT to generate educational patient pamphlets for each procedure using identical commands. Subsequently, two independent radiologists assessed the content, quality, and accuracy of the pamphlets. The review focused on identifying potential errors, inaccuracies, the consistency of pamphlets.
In a thorough analysis of the education pamphlets, we identified shortcomings in 30% (30/100) of pamphlets, with a total of 34 specific inaccuracies, including missing information about sedation for the procedure (10/34), inaccuracies related to specific procedural-related complications (8/34). A key-word co-occurrence network showed consistent themes within each group of pamphlets, while a line-by-line comparison at the level of users and across different procedures showed statistically significant inconsistencies (P < 0.001).
ChatGPT-generated education pamphlets demonstrated potential clinical relevance and fairly consistent terminology; however, the pamphlets were not entirely accurate and exhibited some shortcomings and inter-user structural variabilities. To ensure patient safety, future improvements and refinements in large language models are warranted, while maintaining human supervision and expert validation.</abstract><cop>United States</cop><pub>Elsevier Inc</pub><pmid>38839458</pmid><doi>10.1016/j.acra.2024.05.024</doi><tpages>6</tpages><orcidid>https://orcid.org/0000-0001-8769-0224</orcidid><oa>free_for_read</oa></addata></record> |
| fulltext | fulltext |
| identifier | ISSN: 1076-6332 |
| ispartof | Academic radiology, 2024-11, Vol.31 (11), p.4548-4553 |
| issn | 1076-6332 1878-4046 1878-4046 |
| language | eng |
| recordid | cdi_proquest_miscellaneous_3065276698 |
| source | ScienceDirect Journals (5 years ago - present) |
| subjects | Chat GPT Co-occurrence network graph Education Interventional radiology Large language models |
| title | Evaluation of ChatGPT-Generated Educational Patient Pamphlets for Common Interventional Radiology Procedures |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-06T15%3A46%3A20IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_cross&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Evaluation%20of%20ChatGPT-Generated%20Educational%20Patient%20Pamphlets%20for%20Common%20Interventional%20Radiology%C2%A0Procedures&rft.jtitle=Academic%20radiology&rft.au=Kooraki,%20Soheil&rft.date=2024-11-01&rft.volume=31&rft.issue=11&rft.spage=4548&rft.epage=4553&rft.pages=4548-4553&rft.issn=1076-6332&rft.eissn=1878-4046&rft_id=info:doi/10.1016/j.acra.2024.05.024&rft_dat=%3Cproquest_cross%3E3065276698%3C/proquest_cross%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=3065276698&rft_id=info:pmid/38839458&rft_els_id=S1076633224003076&rfr_iscdi=true |