ChatGPT Conquers the Saudi Medical Licensing Exam: Exploring the Accuracy of Artificial Intelligence in Medical Knowledge Assessment and Implications for Modern Medical Education
BackgroundThe application of artificial intelligence (AI) in education is undergoing rapid advancements, with models such as ChatGPT-4 showing potential in medical education. This study aims to evaluate the proficiency of ChatGPT-4 in answering Saudi Medical Licensing Exam (SMLE) questions.Methodolo...
Gespeichert in:
| Veröffentlicht in: | Curēus (Palo Alto, CA) CA), 2023-09, Vol.15 (9), p.e45043-e45043 |
|---|---|
| Hauptverfasser: | , , , , , , , , |
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | e45043 |
|---|---|
| container_issue | 9 |
| container_start_page | e45043 |
| container_title | Curēus (Palo Alto, CA) |
| container_volume | 15 |
| creator | Aljindan, Fahad K Al Qurashi, Abdullah A Albalawi, Ibrahim Abdullah S Alanazi, Abeer Mohammed M Aljuhani, Hussam Abdulkhaliq M Falah Almutairi, Faisal Aldamigh, Omar A Halawani, Ibrahim R K. Zino Alarki, Subhi M |
| description | BackgroundThe application of artificial intelligence (AI) in education is undergoing rapid advancements, with models such as ChatGPT-4 showing potential in medical education. This study aims to evaluate the proficiency of ChatGPT-4 in answering Saudi Medical Licensing Exam (SMLE) questions.MethodologyA dataset of 220 questions across four medical disciplines was used. The model was trained using a specific code to answer the questions accurately, and its performance was assessed using key performance indicators, difficulty level, and exam sections.ResultsChatGPT-4 demonstrated an overall accuracy of 88.6%. It showed high proficiency with Easy and Average questions, but accuracy decreased for Hard questions. Performance was consistent across all disciplines, indicating a broad knowledge base. However, an error analysis revealed areas for further refinement, particularly with category (Option) A questions across all sections.ConclusionsThis study underscores the potential of ChatGPT-4 as an AI-assisted tool in medical education, demonstrating high proficiency in answering SMLE questions. Future research is recommended to expand the scope of training and evaluation as well as to enhance the model’s performance on complex clinical questions. |
| doi_str_mv | 10.7759/cureus.45043 |
| format | Article |
| fullrecord | <record><control><sourceid>proquest_pubme</sourceid><recordid>TN_cdi_pubmedcentral_primary_oai_pubmedcentral_nih_gov_10566535</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2877388417</sourcerecordid><originalsourceid>FETCH-LOGICAL-c320t-39675e32301842c5dea28a383cdc22bf6830328cee2c7fca05cf0077aed347c13</originalsourceid><addsrcrecordid>eNpdks1OGzEUhUdVKxVRdn0AS910QcBjj8dONyiKUogIAql0bZnrO4nRjJ3anra8Fk-I0yBou_GP7nePfY9OVX2s6YmUYnoKY8QxnTSCNvxNdcDqVk1UrZq3f53fV0cp3VNKayoZlfSgepxvTD6_uSXz4H-MGBPJGyTfzGgduULrwPRk5QB9cn5NFr_N8KWs2z7E3X3HzqC8bOCBhI7MYnadA1ealj5j37s1ekDi_IvYpQ-_erTr0pgSpjSgz8R4S5bDti9EdsEn0oVIroLF-Nq4sOO--qF615k-4dHzflh9_7q4nV9MVtfny_lsNQHOaJ7waSsFcsZpGZyBsGiYMlxxsMDYXdcqTjlTgMhAdmCogI5SKQ1a3kio-WF1ttfdjncD2uJBjqbX2-gGEx90ME7_W_Fuo9fhp66paFvBRVH4_KwQQzE3ZT24BMUW4zGMSTMlJVeqqWVBP_2H3ocx-jJfoVQjGJsKWqjjPQUxpBSxe_lNTfUuBXqfAv0nBfwJ4Hqp3A</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2884522950</pqid></control><display><type>article</type><title>ChatGPT Conquers the Saudi Medical Licensing Exam: Exploring the Accuracy of Artificial Intelligence in Medical Knowledge Assessment and Implications for Modern Medical Education</title><source>PubMed Central Open Access</source><source>PubMed Central</source><creator>Aljindan, Fahad K ; Al Qurashi, Abdullah A ; Albalawi, Ibrahim Abdullah S ; Alanazi, Abeer Mohammed M ; Aljuhani, Hussam Abdulkhaliq M ; Falah Almutairi, Faisal ; Aldamigh, Omar A ; Halawani, Ibrahim R ; K. Zino Alarki, Subhi M</creator><creatorcontrib>Aljindan, Fahad K ; Al Qurashi, Abdullah A ; Albalawi, Ibrahim Abdullah S ; Alanazi, Abeer Mohammed M ; Aljuhani, Hussam Abdulkhaliq M ; Falah Almutairi, Faisal ; Aldamigh, Omar A ; Halawani, Ibrahim R ; K. Zino Alarki, Subhi M</creatorcontrib><description>BackgroundThe application of artificial intelligence (AI) in education is undergoing rapid advancements, with models such as ChatGPT-4 showing potential in medical education. This study aims to evaluate the proficiency of ChatGPT-4 in answering Saudi Medical Licensing Exam (SMLE) questions.MethodologyA dataset of 220 questions across four medical disciplines was used. The model was trained using a specific code to answer the questions accurately, and its performance was assessed using key performance indicators, difficulty level, and exam sections.ResultsChatGPT-4 demonstrated an overall accuracy of 88.6%. It showed high proficiency with Easy and Average questions, but accuracy decreased for Hard questions. Performance was consistent across all disciplines, indicating a broad knowledge base. However, an error analysis revealed areas for further refinement, particularly with category (Option) A questions across all sections.ConclusionsThis study underscores the potential of ChatGPT-4 as an AI-assisted tool in medical education, demonstrating high proficiency in answering SMLE questions. Future research is recommended to expand the scope of training and evaluation as well as to enhance the model’s performance on complex clinical questions.</description><identifier>ISSN: 2168-8184</identifier><identifier>EISSN: 2168-8184</identifier><identifier>DOI: 10.7759/cureus.45043</identifier><language>eng</language><publisher>Palo Alto: Cureus Inc</publisher><subject>Accuracy ; Artificial intelligence ; Business metrics ; Chatbots ; Data collection ; Datasets ; Gynecology ; Healthcare Technology ; Licenses ; Medical Education ; Medicine ; Obstetrics ; Pediatrics ; Performance evaluation ; Quality Improvement ; Surgery</subject><ispartof>Curēus (Palo Alto, CA), 2023-09, Vol.15 (9), p.e45043-e45043</ispartof><rights>Copyright © 2023, Aljindan et al. This work is published under https://creativecommons.org/licenses/by/3.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><rights>Copyright © 2023, Aljindan et al. 2023 Aljindan et al.</rights><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c320t-39675e32301842c5dea28a383cdc22bf6830328cee2c7fca05cf0077aed347c13</citedby><cites>FETCH-LOGICAL-c320t-39675e32301842c5dea28a383cdc22bf6830328cee2c7fca05cf0077aed347c13</cites></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktopdf>$$Uhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC10566535/pdf/$$EPDF$$P50$$Gpubmedcentral$$Hfree_for_read</linktopdf><linktohtml>$$Uhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC10566535/$$EHTML$$P50$$Gpubmedcentral$$Hfree_for_read</linktohtml><link.rule.ids>230,315,729,782,786,887,27931,27932,53798,53800</link.rule.ids></links><search><creatorcontrib>Aljindan, Fahad K</creatorcontrib><creatorcontrib>Al Qurashi, Abdullah A</creatorcontrib><creatorcontrib>Albalawi, Ibrahim Abdullah S</creatorcontrib><creatorcontrib>Alanazi, Abeer Mohammed M</creatorcontrib><creatorcontrib>Aljuhani, Hussam Abdulkhaliq M</creatorcontrib><creatorcontrib>Falah Almutairi, Faisal</creatorcontrib><creatorcontrib>Aldamigh, Omar A</creatorcontrib><creatorcontrib>Halawani, Ibrahim R</creatorcontrib><creatorcontrib>K. Zino Alarki, Subhi M</creatorcontrib><title>ChatGPT Conquers the Saudi Medical Licensing Exam: Exploring the Accuracy of Artificial Intelligence in Medical Knowledge Assessment and Implications for Modern Medical Education</title><title>Curēus (Palo Alto, CA)</title><description>BackgroundThe application of artificial intelligence (AI) in education is undergoing rapid advancements, with models such as ChatGPT-4 showing potential in medical education. This study aims to evaluate the proficiency of ChatGPT-4 in answering Saudi Medical Licensing Exam (SMLE) questions.MethodologyA dataset of 220 questions across four medical disciplines was used. The model was trained using a specific code to answer the questions accurately, and its performance was assessed using key performance indicators, difficulty level, and exam sections.ResultsChatGPT-4 demonstrated an overall accuracy of 88.6%. It showed high proficiency with Easy and Average questions, but accuracy decreased for Hard questions. Performance was consistent across all disciplines, indicating a broad knowledge base. However, an error analysis revealed areas for further refinement, particularly with category (Option) A questions across all sections.ConclusionsThis study underscores the potential of ChatGPT-4 as an AI-assisted tool in medical education, demonstrating high proficiency in answering SMLE questions. Future research is recommended to expand the scope of training and evaluation as well as to enhance the model’s performance on complex clinical questions.</description><subject>Accuracy</subject><subject>Artificial intelligence</subject><subject>Business metrics</subject><subject>Chatbots</subject><subject>Data collection</subject><subject>Datasets</subject><subject>Gynecology</subject><subject>Healthcare Technology</subject><subject>Licenses</subject><subject>Medical Education</subject><subject>Medicine</subject><subject>Obstetrics</subject><subject>Pediatrics</subject><subject>Performance evaluation</subject><subject>Quality Improvement</subject><subject>Surgery</subject><issn>2168-8184</issn><issn>2168-8184</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2023</creationdate><recordtype>article</recordtype><sourceid>ABUWG</sourceid><sourceid>AFKRA</sourceid><sourceid>AZQEC</sourceid><sourceid>BENPR</sourceid><sourceid>CCPQU</sourceid><sourceid>DWQXO</sourceid><recordid>eNpdks1OGzEUhUdVKxVRdn0AS910QcBjj8dONyiKUogIAql0bZnrO4nRjJ3anra8Fk-I0yBou_GP7nePfY9OVX2s6YmUYnoKY8QxnTSCNvxNdcDqVk1UrZq3f53fV0cp3VNKayoZlfSgepxvTD6_uSXz4H-MGBPJGyTfzGgduULrwPRk5QB9cn5NFr_N8KWs2z7E3X3HzqC8bOCBhI7MYnadA1ealj5j37s1ekDi_IvYpQ-_erTr0pgSpjSgz8R4S5bDti9EdsEn0oVIroLF-Nq4sOO--qF615k-4dHzflh9_7q4nV9MVtfny_lsNQHOaJ7waSsFcsZpGZyBsGiYMlxxsMDYXdcqTjlTgMhAdmCogI5SKQ1a3kio-WF1ttfdjncD2uJBjqbX2-gGEx90ME7_W_Fuo9fhp66paFvBRVH4_KwQQzE3ZT24BMUW4zGMSTMlJVeqqWVBP_2H3ocx-jJfoVQjGJsKWqjjPQUxpBSxe_lNTfUuBXqfAv0nBfwJ4Hqp3A</recordid><startdate>20230911</startdate><enddate>20230911</enddate><creator>Aljindan, Fahad K</creator><creator>Al Qurashi, Abdullah A</creator><creator>Albalawi, Ibrahim Abdullah S</creator><creator>Alanazi, Abeer Mohammed M</creator><creator>Aljuhani, Hussam Abdulkhaliq M</creator><creator>Falah Almutairi, Faisal</creator><creator>Aldamigh, Omar A</creator><creator>Halawani, Ibrahim R</creator><creator>K. Zino Alarki, Subhi M</creator><general>Cureus Inc</general><general>Cureus</general><scope>AAYXX</scope><scope>CITATION</scope><scope>3V.</scope><scope>7X7</scope><scope>7XB</scope><scope>8FI</scope><scope>8FJ</scope><scope>8FK</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>FYUFA</scope><scope>GHDGH</scope><scope>K9.</scope><scope>M0S</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>7X8</scope><scope>5PM</scope></search><sort><creationdate>20230911</creationdate><title>ChatGPT Conquers the Saudi Medical Licensing Exam: Exploring the Accuracy of Artificial Intelligence in Medical Knowledge Assessment and Implications for Modern Medical Education</title><author>Aljindan, Fahad K ; Al Qurashi, Abdullah A ; Albalawi, Ibrahim Abdullah S ; Alanazi, Abeer Mohammed M ; Aljuhani, Hussam Abdulkhaliq M ; Falah Almutairi, Faisal ; Aldamigh, Omar A ; Halawani, Ibrahim R ; K. Zino Alarki, Subhi M</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c320t-39675e32301842c5dea28a383cdc22bf6830328cee2c7fca05cf0077aed347c13</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2023</creationdate><topic>Accuracy</topic><topic>Artificial intelligence</topic><topic>Business metrics</topic><topic>Chatbots</topic><topic>Data collection</topic><topic>Datasets</topic><topic>Gynecology</topic><topic>Healthcare Technology</topic><topic>Licenses</topic><topic>Medical Education</topic><topic>Medicine</topic><topic>Obstetrics</topic><topic>Pediatrics</topic><topic>Performance evaluation</topic><topic>Quality Improvement</topic><topic>Surgery</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Aljindan, Fahad K</creatorcontrib><creatorcontrib>Al Qurashi, Abdullah A</creatorcontrib><creatorcontrib>Albalawi, Ibrahim Abdullah S</creatorcontrib><creatorcontrib>Alanazi, Abeer Mohammed M</creatorcontrib><creatorcontrib>Aljuhani, Hussam Abdulkhaliq M</creatorcontrib><creatorcontrib>Falah Almutairi, Faisal</creatorcontrib><creatorcontrib>Aldamigh, Omar A</creatorcontrib><creatorcontrib>Halawani, Ibrahim R</creatorcontrib><creatorcontrib>K. Zino Alarki, Subhi M</creatorcontrib><collection>CrossRef</collection><collection>ProQuest Central (Corporate)</collection><collection>Health & Medical Collection</collection><collection>ProQuest Central (purchase pre-March 2016)</collection><collection>Hospital Premium Collection</collection><collection>Hospital Premium Collection (Alumni Edition)</collection><collection>ProQuest Central (Alumni) (purchase pre-March 2016)</collection><collection>ProQuest Central (Alumni Edition)</collection><collection>ProQuest Central UK/Ireland</collection><collection>ProQuest Central Essentials</collection><collection>ProQuest Central</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central Korea</collection><collection>Health Research Premium Collection</collection><collection>Health Research Premium Collection (Alumni)</collection><collection>ProQuest Health & Medical Complete (Alumni)</collection><collection>Health & Medical Collection (Alumni Edition)</collection><collection>Access via ProQuest (Open Access)</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>MEDLINE - Academic</collection><collection>PubMed Central (Full Participant titles)</collection><jtitle>Curēus (Palo Alto, CA)</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Aljindan, Fahad K</au><au>Al Qurashi, Abdullah A</au><au>Albalawi, Ibrahim Abdullah S</au><au>Alanazi, Abeer Mohammed M</au><au>Aljuhani, Hussam Abdulkhaliq M</au><au>Falah Almutairi, Faisal</au><au>Aldamigh, Omar A</au><au>Halawani, Ibrahim R</au><au>K. Zino Alarki, Subhi M</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>ChatGPT Conquers the Saudi Medical Licensing Exam: Exploring the Accuracy of Artificial Intelligence in Medical Knowledge Assessment and Implications for Modern Medical Education</atitle><jtitle>Curēus (Palo Alto, CA)</jtitle><date>2023-09-11</date><risdate>2023</risdate><volume>15</volume><issue>9</issue><spage>e45043</spage><epage>e45043</epage><pages>e45043-e45043</pages><issn>2168-8184</issn><eissn>2168-8184</eissn><abstract>BackgroundThe application of artificial intelligence (AI) in education is undergoing rapid advancements, with models such as ChatGPT-4 showing potential in medical education. This study aims to evaluate the proficiency of ChatGPT-4 in answering Saudi Medical Licensing Exam (SMLE) questions.MethodologyA dataset of 220 questions across four medical disciplines was used. The model was trained using a specific code to answer the questions accurately, and its performance was assessed using key performance indicators, difficulty level, and exam sections.ResultsChatGPT-4 demonstrated an overall accuracy of 88.6%. It showed high proficiency with Easy and Average questions, but accuracy decreased for Hard questions. Performance was consistent across all disciplines, indicating a broad knowledge base. However, an error analysis revealed areas for further refinement, particularly with category (Option) A questions across all sections.ConclusionsThis study underscores the potential of ChatGPT-4 as an AI-assisted tool in medical education, demonstrating high proficiency in answering SMLE questions. Future research is recommended to expand the scope of training and evaluation as well as to enhance the model’s performance on complex clinical questions.</abstract><cop>Palo Alto</cop><pub>Cureus Inc</pub><doi>10.7759/cureus.45043</doi><oa>free_for_read</oa></addata></record> |
| fulltext | fulltext |
| identifier | ISSN: 2168-8184 |
| ispartof | Curēus (Palo Alto, CA), 2023-09, Vol.15 (9), p.e45043-e45043 |
| issn | 2168-8184 2168-8184 |
| language | eng |
| recordid | cdi_pubmedcentral_primary_oai_pubmedcentral_nih_gov_10566535 |
| source | PubMed Central Open Access; PubMed Central |
| subjects | Accuracy Artificial intelligence Business metrics Chatbots Data collection Datasets Gynecology Healthcare Technology Licenses Medical Education Medicine Obstetrics Pediatrics Performance evaluation Quality Improvement Surgery |
| title | ChatGPT Conquers the Saudi Medical Licensing Exam: Exploring the Accuracy of Artificial Intelligence in Medical Knowledge Assessment and Implications for Modern Medical Education |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-05T02%3A59%3A56IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_pubme&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=ChatGPT%20Conquers%20the%20Saudi%20Medical%20Licensing%20Exam:%20Exploring%20the%20Accuracy%20of%20Artificial%20Intelligence%20in%20Medical%20Knowledge%20Assessment%20and%20Implications%20for%20Modern%20Medical%20Education&rft.jtitle=Cur%C4%93us%20(Palo%20Alto,%20CA)&rft.au=Aljindan,%20Fahad%20K&rft.date=2023-09-11&rft.volume=15&rft.issue=9&rft.spage=e45043&rft.epage=e45043&rft.pages=e45043-e45043&rft.issn=2168-8184&rft.eissn=2168-8184&rft_id=info:doi/10.7759/cureus.45043&rft_dat=%3Cproquest_pubme%3E2877388417%3C/proquest_pubme%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2884522950&rft_id=info:pmid/&rfr_iscdi=true |