ChatGPT- versus human-generated answers to frequently asked questions about diabetes: A Turing test-inspired survey among employees of a Danish diabetes center
Large language models have received enormous attention recently with some studies demonstrating their potential clinical value, despite not being trained specifically for this domain. We aimed to investigate whether ChatGPT, a language model optimized for dialogue, can answer frequently asked questi...
Gespeichert in:
| Veröffentlicht in: | PloS one 2023-08, Vol.18 (8), p.e0290773 |
|---|---|
| Hauptverfasser: | , , , , , , |
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | |
|---|---|
| container_issue | 8 |
| container_start_page | e0290773 |
| container_title | PloS one |
| container_volume | 18 |
| creator | Hulman, Adam Dollerup, Ole Lindgård Mortensen, Jesper Friis Fenech, Matthew E Norman, Kasper Støvring, Henrik Hansen, Troels Krarup |
| description | Large language models have received enormous attention recently with some studies demonstrating their potential clinical value, despite not being trained specifically for this domain. We aimed to investigate whether ChatGPT, a language model optimized for dialogue, can answer frequently asked questions about diabetes. We conducted a closed e-survey among employees of a large Danish diabetes center. The study design was inspired by the Turing test and non-inferiority trials. Our survey included ten questions with two answers each. One of these was written by a human expert, while the other was generated by ChatGPT. Participants had the task to identify the ChatGPT-generated answer. Data was analyzed at the question-level using logistic regression with robust variance estimation with clustering at participant level. In secondary analyses, we investigated the effect of participant characteristics on the outcome. A 55% non-inferiority margin was pre-defined based on precision simulations and had been published as part of the study protocol before data collection began. Among 311 invited individuals, 183 participated in the survey (59% response rate). 64% had heard of ChatGPT before, and 19% had tried it. Overall, participants could identify ChatGPT-generated answers 59.5% (95% CI: 57.0, 62.0) of the time, which was outside of the non-inferiority zone. Among participant characteristics, previous ChatGPT use had the strongest association with the outcome (odds ratio: 1.52 (1.16, 2.00), p = 0.003). Previous users answered 67.4% (61.7, 72.7) of the questions correctly, versus non-users' 57.6% (54.9, 60.3). Participants could distinguish between ChatGPT-generated and human-written answers somewhat better than flipping a fair coin, which was against our initial hypothesis. Rigorously planned studies are needed to elucidate the risks and benefits of integrating such technologies in routine clinical practice. |
| doi_str_mv | 10.1371/journal.pone.0290773 |
| format | Article |
| fullrecord | <record><control><sourceid>gale_plos_</sourceid><recordid>TN_cdi_plos_journals_2859587738</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><galeid>A762899228</galeid><doaj_id>oai_doaj_org_article_09f23ef75e4d4397984f73b5a86baba8</doaj_id><sourcerecordid>A762899228</sourcerecordid><originalsourceid>FETCH-LOGICAL-c6083-57a32a3b0e366c84d3704f95444b059b03100a5d0b44db84e99050d3ecb783003</originalsourceid><addsrcrecordid>eNqNk8GO0zAQhiMEYpeFN0BgCQnBIcWJncThgqoCS6WVFkHhak2SSeslsYudFPo0vCpOm61atAeUQ2LP9__jGWeC4GlEJxHLojc3prcamsnaaJzQOKdZxu4F51HO4jCNKbt_9H0WPHLuhtKEiTR9GJyxLE0iJqLz4M9sBd3l50VINmhd78iqb0GHS9RoocOKgHa_fIR0htQWf_aou2ZLwP3wMb9ynTLaEShM35FKQYEdurdkSha9VXpJ_KoLlXZrZb3A9XaDXt0aH8J23ZgtoiOmJkDeg1ZudfAgpc-E9nHwoIbG4ZPxfRF8-_hhMfsUXl1fzmfTq7BMqWBhkgGLgRUUWZqWglcso7zOE855QZO8oCyiFJKKFpxXheCY5zShFcOyyASjlF0Ez_e-_kxOjr11MhZJngjfWeGJ-Z6oDNzItVUt2K00oORuw9ilBNupskFJ8zpmWGcJ8oqzPMsFrzNWJCDSAgoYvN6N2fqixWoo1UJzYnoa0Woll2YjI8ozKvLcO7waHazZXYNslSuxaUCj6YeDp5TTJBVDaS_-Qe8ub6SW4CtQujY-cTmYymmWxj5nHA_U5A7KPxW2qvR_Yq38_ong9YnAMx3-7pbQOyfnX7_8P3v9_ZR9ecSuEJpu5UzT737HU5DvwdIa5yzWhy5HVA6DdNsNOQySHAfJy54d39BBdDs57C_2nhoX</addsrcrecordid><sourcetype>Open Website</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2859587738</pqid></control><display><type>article</type><title>ChatGPT- versus human-generated answers to frequently asked questions about diabetes: A Turing test-inspired survey among employees of a Danish diabetes center</title><source>MEDLINE</source><source>DOAJ Directory of Open Access Journals</source><source>EZB-FREE-00999 freely available EZB journals</source><source>PubMed Central</source><source>Free Full-Text Journals in Chemistry</source><source>Public Library of Science (PLoS)</source><creator>Hulman, Adam ; Dollerup, Ole Lindgård ; Mortensen, Jesper Friis ; Fenech, Matthew E ; Norman, Kasper ; Støvring, Henrik ; Hansen, Troels Krarup</creator><creatorcontrib>Hulman, Adam ; Dollerup, Ole Lindgård ; Mortensen, Jesper Friis ; Fenech, Matthew E ; Norman, Kasper ; Støvring, Henrik ; Hansen, Troels Krarup</creatorcontrib><description>Large language models have received enormous attention recently with some studies demonstrating their potential clinical value, despite not being trained specifically for this domain. We aimed to investigate whether ChatGPT, a language model optimized for dialogue, can answer frequently asked questions about diabetes. We conducted a closed e-survey among employees of a large Danish diabetes center. The study design was inspired by the Turing test and non-inferiority trials. Our survey included ten questions with two answers each. One of these was written by a human expert, while the other was generated by ChatGPT. Participants had the task to identify the ChatGPT-generated answer. Data was analyzed at the question-level using logistic regression with robust variance estimation with clustering at participant level. In secondary analyses, we investigated the effect of participant characteristics on the outcome. A 55% non-inferiority margin was pre-defined based on precision simulations and had been published as part of the study protocol before data collection began. Among 311 invited individuals, 183 participated in the survey (59% response rate). 64% had heard of ChatGPT before, and 19% had tried it. Overall, participants could identify ChatGPT-generated answers 59.5% (95% CI: 57.0, 62.0) of the time, which was outside of the non-inferiority zone. Among participant characteristics, previous ChatGPT use had the strongest association with the outcome (odds ratio: 1.52 (1.16, 2.00), p = 0.003). Previous users answered 67.4% (61.7, 72.7) of the questions correctly, versus non-users' 57.6% (54.9, 60.3). Participants could distinguish between ChatGPT-generated and human-written answers somewhat better than flipping a fair coin, which was against our initial hypothesis. Rigorously planned studies are needed to elucidate the risks and benefits of integrating such technologies in routine clinical practice.</description><identifier>ISSN: 1932-6203</identifier><identifier>EISSN: 1932-6203</identifier><identifier>DOI: 10.1371/journal.pone.0290773</identifier><identifier>PMID: 37651381</identifier><language>eng</language><publisher>United States: Public Library of Science</publisher><subject>Artificial intelligence ; Beliefs, opinions and attitudes ; Biology and Life Sciences ; Chatbots ; Clinical trials ; Cluster Analysis ; Clustering ; Computation ; Computer and Information Sciences ; Data Collection ; Denmark - epidemiology ; Diabetes ; Diabetes mellitus ; Diabetes Mellitus - diagnosis ; Diabetes Mellitus - epidemiology ; Employees ; Evaluation ; Forecasts and trends ; Health care industry ; Health risks ; Health surveys ; Humans ; Innovations ; Language ; Large language models ; Logic ; Medical personnel ; Medicine and Health Sciences ; Patients ; Polls & surveys ; Power ; Professional ethics ; Professionals ; Questions ; Regression analysis ; Robustness (mathematics) ; Social Sciences ; Surveys ; Technology application</subject><ispartof>PloS one, 2023-08, Vol.18 (8), p.e0290773</ispartof><rights>Copyright: © 2023 Hulman et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.</rights><rights>COPYRIGHT 2023 Public Library of Science</rights><rights>2023 Hulman et al. This is an open access article distributed under the terms of the Creative Commons Attribution License: http://creativecommons.org/licenses/by/4.0/ (the “License”), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><rights>2023 Hulman et al 2023 Hulman et al</rights><rights>2023 Hulman et al. This is an open access article distributed under the terms of the Creative Commons Attribution License: http://creativecommons.org/licenses/by/4.0/ (the “License”), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c6083-57a32a3b0e366c84d3704f95444b059b03100a5d0b44db84e99050d3ecb783003</citedby><cites>FETCH-LOGICAL-c6083-57a32a3b0e366c84d3704f95444b059b03100a5d0b44db84e99050d3ecb783003</cites><orcidid>0000-0001-6924-7114 ; 0000-0002-5821-2351 ; 0000-0002-0708-2868 ; 0000-0002-3969-1000 ; 0000-0001-7683-0906 ; 0000-0001-9970-0215</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktopdf>$$Uhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC10470899/pdf/$$EPDF$$P50$$Gpubmedcentral$$Hfree_for_read</linktopdf><linktohtml>$$Uhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC10470899/$$EHTML$$P50$$Gpubmedcentral$$Hfree_for_read</linktohtml><link.rule.ids>230,314,723,776,780,860,881,2096,2915,23845,27901,27902,53766,53768,79342,79343</link.rule.ids><backlink>$$Uhttps://www.ncbi.nlm.nih.gov/pubmed/37651381$$D View this record in MEDLINE/PubMed$$Hfree_for_read</backlink></links><search><creatorcontrib>Hulman, Adam</creatorcontrib><creatorcontrib>Dollerup, Ole Lindgård</creatorcontrib><creatorcontrib>Mortensen, Jesper Friis</creatorcontrib><creatorcontrib>Fenech, Matthew E</creatorcontrib><creatorcontrib>Norman, Kasper</creatorcontrib><creatorcontrib>Støvring, Henrik</creatorcontrib><creatorcontrib>Hansen, Troels Krarup</creatorcontrib><title>ChatGPT- versus human-generated answers to frequently asked questions about diabetes: A Turing test-inspired survey among employees of a Danish diabetes center</title><title>PloS one</title><addtitle>PLoS One</addtitle><description>Large language models have received enormous attention recently with some studies demonstrating their potential clinical value, despite not being trained specifically for this domain. We aimed to investigate whether ChatGPT, a language model optimized for dialogue, can answer frequently asked questions about diabetes. We conducted a closed e-survey among employees of a large Danish diabetes center. The study design was inspired by the Turing test and non-inferiority trials. Our survey included ten questions with two answers each. One of these was written by a human expert, while the other was generated by ChatGPT. Participants had the task to identify the ChatGPT-generated answer. Data was analyzed at the question-level using logistic regression with robust variance estimation with clustering at participant level. In secondary analyses, we investigated the effect of participant characteristics on the outcome. A 55% non-inferiority margin was pre-defined based on precision simulations and had been published as part of the study protocol before data collection began. Among 311 invited individuals, 183 participated in the survey (59% response rate). 64% had heard of ChatGPT before, and 19% had tried it. Overall, participants could identify ChatGPT-generated answers 59.5% (95% CI: 57.0, 62.0) of the time, which was outside of the non-inferiority zone. Among participant characteristics, previous ChatGPT use had the strongest association with the outcome (odds ratio: 1.52 (1.16, 2.00), p = 0.003). Previous users answered 67.4% (61.7, 72.7) of the questions correctly, versus non-users' 57.6% (54.9, 60.3). Participants could distinguish between ChatGPT-generated and human-written answers somewhat better than flipping a fair coin, which was against our initial hypothesis. Rigorously planned studies are needed to elucidate the risks and benefits of integrating such technologies in routine clinical practice.</description><subject>Artificial intelligence</subject><subject>Beliefs, opinions and attitudes</subject><subject>Biology and Life Sciences</subject><subject>Chatbots</subject><subject>Clinical trials</subject><subject>Cluster Analysis</subject><subject>Clustering</subject><subject>Computation</subject><subject>Computer and Information Sciences</subject><subject>Data Collection</subject><subject>Denmark - epidemiology</subject><subject>Diabetes</subject><subject>Diabetes mellitus</subject><subject>Diabetes Mellitus - diagnosis</subject><subject>Diabetes Mellitus - epidemiology</subject><subject>Employees</subject><subject>Evaluation</subject><subject>Forecasts and trends</subject><subject>Health care industry</subject><subject>Health risks</subject><subject>Health surveys</subject><subject>Humans</subject><subject>Innovations</subject><subject>Language</subject><subject>Large language models</subject><subject>Logic</subject><subject>Medical personnel</subject><subject>Medicine and Health Sciences</subject><subject>Patients</subject><subject>Polls & surveys</subject><subject>Power</subject><subject>Professional ethics</subject><subject>Professionals</subject><subject>Questions</subject><subject>Regression analysis</subject><subject>Robustness (mathematics)</subject><subject>Social Sciences</subject><subject>Surveys</subject><subject>Technology application</subject><issn>1932-6203</issn><issn>1932-6203</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2023</creationdate><recordtype>article</recordtype><sourceid>EIF</sourceid><sourceid>BENPR</sourceid><sourceid>DOA</sourceid><recordid>eNqNk8GO0zAQhiMEYpeFN0BgCQnBIcWJncThgqoCS6WVFkHhak2SSeslsYudFPo0vCpOm61atAeUQ2LP9__jGWeC4GlEJxHLojc3prcamsnaaJzQOKdZxu4F51HO4jCNKbt_9H0WPHLuhtKEiTR9GJyxLE0iJqLz4M9sBd3l50VINmhd78iqb0GHS9RoocOKgHa_fIR0htQWf_aou2ZLwP3wMb9ynTLaEShM35FKQYEdurdkSha9VXpJ_KoLlXZrZb3A9XaDXt0aH8J23ZgtoiOmJkDeg1ZudfAgpc-E9nHwoIbG4ZPxfRF8-_hhMfsUXl1fzmfTq7BMqWBhkgGLgRUUWZqWglcso7zOE855QZO8oCyiFJKKFpxXheCY5zShFcOyyASjlF0Ez_e-_kxOjr11MhZJngjfWeGJ-Z6oDNzItVUt2K00oORuw9ilBNupskFJ8zpmWGcJ8oqzPMsFrzNWJCDSAgoYvN6N2fqixWoo1UJzYnoa0Woll2YjI8ozKvLcO7waHazZXYNslSuxaUCj6YeDp5TTJBVDaS_-Qe8ub6SW4CtQujY-cTmYymmWxj5nHA_U5A7KPxW2qvR_Yq38_ong9YnAMx3-7pbQOyfnX7_8P3v9_ZR9ecSuEJpu5UzT737HU5DvwdIa5yzWhy5HVA6DdNsNOQySHAfJy54d39BBdDs57C_2nhoX</recordid><startdate>20230831</startdate><enddate>20230831</enddate><creator>Hulman, Adam</creator><creator>Dollerup, Ole Lindgård</creator><creator>Mortensen, Jesper Friis</creator><creator>Fenech, Matthew E</creator><creator>Norman, Kasper</creator><creator>Støvring, Henrik</creator><creator>Hansen, Troels Krarup</creator><general>Public Library of Science</general><general>Public Library of Science (PLoS)</general><scope>CGR</scope><scope>CUY</scope><scope>CVF</scope><scope>ECM</scope><scope>EIF</scope><scope>NPM</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>IOV</scope><scope>ISR</scope><scope>3V.</scope><scope>7QG</scope><scope>7QL</scope><scope>7QO</scope><scope>7RV</scope><scope>7SN</scope><scope>7SS</scope><scope>7T5</scope><scope>7TG</scope><scope>7TM</scope><scope>7U9</scope><scope>7X2</scope><scope>7X7</scope><scope>7XB</scope><scope>88E</scope><scope>8AO</scope><scope>8C1</scope><scope>8FD</scope><scope>8FE</scope><scope>8FG</scope><scope>8FH</scope><scope>8FI</scope><scope>8FJ</scope><scope>8FK</scope><scope>ABJCF</scope><scope>ABUWG</scope><scope>AEUYN</scope><scope>AFKRA</scope><scope>ARAPS</scope><scope>ATCPS</scope><scope>AZQEC</scope><scope>BBNVY</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>BHPHI</scope><scope>C1K</scope><scope>CCPQU</scope><scope>D1I</scope><scope>DWQXO</scope><scope>FR3</scope><scope>FYUFA</scope><scope>GHDGH</scope><scope>GNUQQ</scope><scope>H94</scope><scope>HCIFZ</scope><scope>K9.</scope><scope>KB.</scope><scope>KB0</scope><scope>KL.</scope><scope>L6V</scope><scope>LK8</scope><scope>M0K</scope><scope>M0S</scope><scope>M1P</scope><scope>M7N</scope><scope>M7P</scope><scope>M7S</scope><scope>NAPCQ</scope><scope>P5Z</scope><scope>P62</scope><scope>P64</scope><scope>PATMY</scope><scope>PDBOC</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>PTHSS</scope><scope>PYCSY</scope><scope>RC3</scope><scope>7X8</scope><scope>5PM</scope><scope>DOA</scope><orcidid>https://orcid.org/0000-0001-6924-7114</orcidid><orcidid>https://orcid.org/0000-0002-5821-2351</orcidid><orcidid>https://orcid.org/0000-0002-0708-2868</orcidid><orcidid>https://orcid.org/0000-0002-3969-1000</orcidid><orcidid>https://orcid.org/0000-0001-7683-0906</orcidid><orcidid>https://orcid.org/0000-0001-9970-0215</orcidid></search><sort><creationdate>20230831</creationdate><title>ChatGPT- versus human-generated answers to frequently asked questions about diabetes: A Turing test-inspired survey among employees of a Danish diabetes center</title><author>Hulman, Adam ; Dollerup, Ole Lindgård ; Mortensen, Jesper Friis ; Fenech, Matthew E ; Norman, Kasper ; Støvring, Henrik ; Hansen, Troels Krarup</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c6083-57a32a3b0e366c84d3704f95444b059b03100a5d0b44db84e99050d3ecb783003</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2023</creationdate><topic>Artificial intelligence</topic><topic>Beliefs, opinions and attitudes</topic><topic>Biology and Life Sciences</topic><topic>Chatbots</topic><topic>Clinical trials</topic><topic>Cluster Analysis</topic><topic>Clustering</topic><topic>Computation</topic><topic>Computer and Information Sciences</topic><topic>Data Collection</topic><topic>Denmark - epidemiology</topic><topic>Diabetes</topic><topic>Diabetes mellitus</topic><topic>Diabetes Mellitus - diagnosis</topic><topic>Diabetes Mellitus - epidemiology</topic><topic>Employees</topic><topic>Evaluation</topic><topic>Forecasts and trends</topic><topic>Health care industry</topic><topic>Health risks</topic><topic>Health surveys</topic><topic>Humans</topic><topic>Innovations</topic><topic>Language</topic><topic>Large language models</topic><topic>Logic</topic><topic>Medical personnel</topic><topic>Medicine and Health Sciences</topic><topic>Patients</topic><topic>Polls & surveys</topic><topic>Power</topic><topic>Professional ethics</topic><topic>Professionals</topic><topic>Questions</topic><topic>Regression analysis</topic><topic>Robustness (mathematics)</topic><topic>Social Sciences</topic><topic>Surveys</topic><topic>Technology application</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Hulman, Adam</creatorcontrib><creatorcontrib>Dollerup, Ole Lindgård</creatorcontrib><creatorcontrib>Mortensen, Jesper Friis</creatorcontrib><creatorcontrib>Fenech, Matthew E</creatorcontrib><creatorcontrib>Norman, Kasper</creatorcontrib><creatorcontrib>Støvring, Henrik</creatorcontrib><creatorcontrib>Hansen, Troels Krarup</creatorcontrib><collection>Medline</collection><collection>MEDLINE</collection><collection>MEDLINE (Ovid)</collection><collection>MEDLINE</collection><collection>MEDLINE</collection><collection>PubMed</collection><collection>CrossRef</collection><collection>Gale In Context: Opposing Viewpoints</collection><collection>Gale In Context: Science</collection><collection>ProQuest Central (Corporate)</collection><collection>Animal Behavior Abstracts</collection><collection>Bacteriology Abstracts (Microbiology B)</collection><collection>Biotechnology Research Abstracts</collection><collection>Nursing & Allied Health Database</collection><collection>Ecology Abstracts</collection><collection>Entomology Abstracts (Full archive)</collection><collection>Immunology Abstracts</collection><collection>Meteorological & Geoastrophysical Abstracts</collection><collection>Nucleic Acids Abstracts</collection><collection>Virology and AIDS Abstracts</collection><collection>Agricultural Science Collection</collection><collection>Health & Medical Collection</collection><collection>ProQuest Central (purchase pre-March 2016)</collection><collection>Medical Database (Alumni Edition)</collection><collection>ProQuest Pharma Collection</collection><collection>Public Health Database</collection><collection>Technology Research Database</collection><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>ProQuest Natural Science Collection</collection><collection>Hospital Premium Collection</collection><collection>Hospital Premium Collection (Alumni Edition)</collection><collection>ProQuest Central (Alumni) (purchase pre-March 2016)</collection><collection>Materials Science & Engineering Collection</collection><collection>ProQuest Central (Alumni Edition)</collection><collection>ProQuest One Sustainability</collection><collection>ProQuest Central UK/Ireland</collection><collection>Advanced Technologies & Aerospace Collection</collection><collection>Agricultural & Environmental Science Collection</collection><collection>ProQuest Central Essentials</collection><collection>Biological Science Collection</collection><collection>ProQuest Central</collection><collection>Technology Collection</collection><collection>Natural Science Collection</collection><collection>Environmental Sciences and Pollution Management</collection><collection>ProQuest One Community College</collection><collection>ProQuest Materials Science Collection</collection><collection>ProQuest Central Korea</collection><collection>Engineering Research Database</collection><collection>Health Research Premium Collection</collection><collection>Health Research Premium Collection (Alumni)</collection><collection>ProQuest Central Student</collection><collection>AIDS and Cancer Research Abstracts</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Health & Medical Complete (Alumni)</collection><collection>Materials Science Database</collection><collection>Nursing & Allied Health Database (Alumni Edition)</collection><collection>Meteorological & Geoastrophysical Abstracts - Academic</collection><collection>ProQuest Engineering Collection</collection><collection>ProQuest Biological Science Collection</collection><collection>Agricultural Science Database</collection><collection>Health & Medical Collection (Alumni Edition)</collection><collection>Medical Database</collection><collection>Algology Mycology and Protozoology Abstracts (Microbiology C)</collection><collection>Biological Science Database</collection><collection>Engineering Database</collection><collection>Nursing & Allied Health Premium</collection><collection>Advanced Technologies & Aerospace Database</collection><collection>ProQuest Advanced Technologies & Aerospace Collection</collection><collection>Biotechnology and BioEngineering Abstracts</collection><collection>Environmental Science Database</collection><collection>Materials Science Collection</collection><collection>Publicly Available Content Database</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>Engineering Collection</collection><collection>Environmental Science Collection</collection><collection>Genetics Abstracts</collection><collection>MEDLINE - Academic</collection><collection>PubMed Central (Full Participant titles)</collection><collection>DOAJ Directory of Open Access Journals</collection><jtitle>PloS one</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Hulman, Adam</au><au>Dollerup, Ole Lindgård</au><au>Mortensen, Jesper Friis</au><au>Fenech, Matthew E</au><au>Norman, Kasper</au><au>Støvring, Henrik</au><au>Hansen, Troels Krarup</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>ChatGPT- versus human-generated answers to frequently asked questions about diabetes: A Turing test-inspired survey among employees of a Danish diabetes center</atitle><jtitle>PloS one</jtitle><addtitle>PLoS One</addtitle><date>2023-08-31</date><risdate>2023</risdate><volume>18</volume><issue>8</issue><spage>e0290773</spage><pages>e0290773-</pages><issn>1932-6203</issn><eissn>1932-6203</eissn><abstract>Large language models have received enormous attention recently with some studies demonstrating their potential clinical value, despite not being trained specifically for this domain. We aimed to investigate whether ChatGPT, a language model optimized for dialogue, can answer frequently asked questions about diabetes. We conducted a closed e-survey among employees of a large Danish diabetes center. The study design was inspired by the Turing test and non-inferiority trials. Our survey included ten questions with two answers each. One of these was written by a human expert, while the other was generated by ChatGPT. Participants had the task to identify the ChatGPT-generated answer. Data was analyzed at the question-level using logistic regression with robust variance estimation with clustering at participant level. In secondary analyses, we investigated the effect of participant characteristics on the outcome. A 55% non-inferiority margin was pre-defined based on precision simulations and had been published as part of the study protocol before data collection began. Among 311 invited individuals, 183 participated in the survey (59% response rate). 64% had heard of ChatGPT before, and 19% had tried it. Overall, participants could identify ChatGPT-generated answers 59.5% (95% CI: 57.0, 62.0) of the time, which was outside of the non-inferiority zone. Among participant characteristics, previous ChatGPT use had the strongest association with the outcome (odds ratio: 1.52 (1.16, 2.00), p = 0.003). Previous users answered 67.4% (61.7, 72.7) of the questions correctly, versus non-users' 57.6% (54.9, 60.3). Participants could distinguish between ChatGPT-generated and human-written answers somewhat better than flipping a fair coin, which was against our initial hypothesis. Rigorously planned studies are needed to elucidate the risks and benefits of integrating such technologies in routine clinical practice.</abstract><cop>United States</cop><pub>Public Library of Science</pub><pmid>37651381</pmid><doi>10.1371/journal.pone.0290773</doi><tpages>e0290773</tpages><orcidid>https://orcid.org/0000-0001-6924-7114</orcidid><orcidid>https://orcid.org/0000-0002-5821-2351</orcidid><orcidid>https://orcid.org/0000-0002-0708-2868</orcidid><orcidid>https://orcid.org/0000-0002-3969-1000</orcidid><orcidid>https://orcid.org/0000-0001-7683-0906</orcidid><orcidid>https://orcid.org/0000-0001-9970-0215</orcidid><oa>free_for_read</oa></addata></record> |
| fulltext | fulltext |
| identifier | ISSN: 1932-6203 |
| ispartof | PloS one, 2023-08, Vol.18 (8), p.e0290773 |
| issn | 1932-6203 1932-6203 |
| language | eng |
| recordid | cdi_plos_journals_2859587738 |
| source | MEDLINE; DOAJ Directory of Open Access Journals; EZB-FREE-00999 freely available EZB journals; PubMed Central; Free Full-Text Journals in Chemistry; Public Library of Science (PLoS) |
| subjects | Artificial intelligence Beliefs, opinions and attitudes Biology and Life Sciences Chatbots Clinical trials Cluster Analysis Clustering Computation Computer and Information Sciences Data Collection Denmark - epidemiology Diabetes Diabetes mellitus Diabetes Mellitus - diagnosis Diabetes Mellitus - epidemiology Employees Evaluation Forecasts and trends Health care industry Health risks Health surveys Humans Innovations Language Large language models Logic Medical personnel Medicine and Health Sciences Patients Polls & surveys Power Professional ethics Professionals Questions Regression analysis Robustness (mathematics) Social Sciences Surveys Technology application |
| title | ChatGPT- versus human-generated answers to frequently asked questions about diabetes: A Turing test-inspired survey among employees of a Danish diabetes center |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-02-10T09%3A19%3A04IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-gale_plos_&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=ChatGPT-%20versus%20human-generated%20answers%20to%20frequently%20asked%20questions%20about%20diabetes:%20A%20Turing%20test-inspired%20survey%20among%20employees%20of%20a%20Danish%20diabetes%20center&rft.jtitle=PloS%20one&rft.au=Hulman,%20Adam&rft.date=2023-08-31&rft.volume=18&rft.issue=8&rft.spage=e0290773&rft.pages=e0290773-&rft.issn=1932-6203&rft.eissn=1932-6203&rft_id=info:doi/10.1371/journal.pone.0290773&rft_dat=%3Cgale_plos_%3EA762899228%3C/gale_plos_%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2859587738&rft_id=info:pmid/37651381&rft_galeid=A762899228&rft_doaj_id=oai_doaj_org_article_09f23ef75e4d4397984f73b5a86baba8&rfr_iscdi=true |