The value of Retrospective and Concurrent Think Aloud in formative usability testing of a physician data query tool
[Display omitted] •Comparison of Concurrent and Retrospective Think Aloud in usability testing of HIT.•Concurrent and Retrospective Think Aloud differ in usability problem detection scope.•Concurrent Think Aloud has higher effectiveness in terms of system redesign.•Retrospective Think Aloud is promo...
Gespeichert in:
| Veröffentlicht in: | Journal of biomedical informatics 2015-06, Vol.55, p.1-10 |
|---|---|
| Hauptverfasser: | , , |
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | 10 |
|---|---|
| container_issue | |
| container_start_page | 1 |
| container_title | Journal of biomedical informatics |
| container_volume | 55 |
| creator | Peute, Linda W.P. de Keizer, Nicolette F. Jaspers, Monique W.M. |
| description | [Display omitted]
•Comparison of Concurrent and Retrospective Think Aloud in usability testing of HIT.•Concurrent and Retrospective Think Aloud differ in usability problem detection scope.•Concurrent Think Aloud has higher effectiveness in terms of system redesign.•Retrospective Think Aloud is promoted for research on expert user customized design.
To compare the performance of the Concurrent (CTA) and Retrospective (RTA) Think Aloud method and to assess their value in a formative usability evaluation of an Intensive Care Registry-physician data query tool designed to support ICU quality improvement processes.
Sixteen representative intensive care physicians participated in the usability evaluation study. Subjects were allocated to either the CTA or RTA method by a matched randomized design. Each subject performed six usability-testing tasks of varying complexity in the query tool in a real-working context. Methods were compared with regard to number and type of problems detected. Verbal protocols of CTA and RTA were analyzed in depth to assess differences in verbal output. Standardized measures were applied to assess thoroughness in usability problem detection weighted per problem severity level and method overall effectiveness in detecting usability problems with regard to the time subjects spent per method.
The usability evaluation of the data query tool revealed a total of 43 unique usability problems that the intensive care physicians encountered. CTA detected unique usability problems with regard to graphics/symbols, navigation issues, error messages, and the organization of information on the query tool’s screens. RTA detected unique issues concerning system match with subjects’ language and applied terminology. The in-depth verbal protocol analysis of CTA provided information on intensive care physicians’ query design strategies. Overall, CTA performed significantly better than RTA in detecting usability problems. CTA usability problem detection effectiveness was 0.80 vs. 0.62 (p |
| doi_str_mv | 10.1016/j.jbi.2015.02.006 |
| format | Article |
| fullrecord | <record><control><sourceid>proquest_cross</sourceid><recordid>TN_cdi_proquest_miscellaneous_1709736604</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><els_id>S1532046415000386</els_id><sourcerecordid>1701490567</sourcerecordid><originalsourceid>FETCH-LOGICAL-c462t-1883cb58ef3dbc18ce9b013e623480e2a4cd342db06e3d69c3a1ffe33674b1f93</originalsourceid><addsrcrecordid>eNqNkcFuEzEQhq2Kqi2FB-CCfOSSxWPvenfFqYpKi1QJqQpny2vPNg4bO9jeSHl7HNL2CJw8h2_-8cxHyAdgFTCQnzfVZnAVZ9BUjFeMyTNyBY3gC1Z37M1rLetL8jalDWMATSMvyCVvWiFAyiuSVmukez3NSMNIHzHHkHZostsj1d7SZfBmjhF9pqu18z_pzRRmS52nY4hb_Yebkx7c5PKBZkzZ-adjlKa79SE547SnVmdNf80YCxHC9I6cj3pK-P75vSY_vt6ulveLh-9335Y3DwtTS54X0HXCDE2Ho7CDgc5gPzAQKLko6yHXtbGi5nZgEoWVvREaxhGFkG09wNiLa_LplLuLoUxPWW1dMjhN2mOYk4KW9a2QktX_g0Lds0a2_0Zl15YP9n1XUDihplw1RRzVLrqtjgcFTB0Nqo0qBtXRoGJcFYOl5-Nz_Dxs0b52vCgrwJcTgOV0e4dRJePQG7QuFnHKBveX-N-I_6ws</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>1687348998</pqid></control><display><type>article</type><title>The value of Retrospective and Concurrent Think Aloud in formative usability testing of a physician data query tool</title><source>MEDLINE</source><source>Elektronische Zeitschriftenbibliothek - Frei zugängliche E-Journals</source><source>Access via ScienceDirect (Elsevier)</source><creator>Peute, Linda W.P. ; de Keizer, Nicolette F. ; Jaspers, Monique W.M.</creator><creatorcontrib>Peute, Linda W.P. ; de Keizer, Nicolette F. ; Jaspers, Monique W.M.</creatorcontrib><description>[Display omitted]
•Comparison of Concurrent and Retrospective Think Aloud in usability testing of HIT.•Concurrent and Retrospective Think Aloud differ in usability problem detection scope.•Concurrent Think Aloud has higher effectiveness in terms of system redesign.•Retrospective Think Aloud is promoted for research on expert user customized design.
To compare the performance of the Concurrent (CTA) and Retrospective (RTA) Think Aloud method and to assess their value in a formative usability evaluation of an Intensive Care Registry-physician data query tool designed to support ICU quality improvement processes.
Sixteen representative intensive care physicians participated in the usability evaluation study. Subjects were allocated to either the CTA or RTA method by a matched randomized design. Each subject performed six usability-testing tasks of varying complexity in the query tool in a real-working context. Methods were compared with regard to number and type of problems detected. Verbal protocols of CTA and RTA were analyzed in depth to assess differences in verbal output. Standardized measures were applied to assess thoroughness in usability problem detection weighted per problem severity level and method overall effectiveness in detecting usability problems with regard to the time subjects spent per method.
The usability evaluation of the data query tool revealed a total of 43 unique usability problems that the intensive care physicians encountered. CTA detected unique usability problems with regard to graphics/symbols, navigation issues, error messages, and the organization of information on the query tool’s screens. RTA detected unique issues concerning system match with subjects’ language and applied terminology. The in-depth verbal protocol analysis of CTA provided information on intensive care physicians’ query design strategies. Overall, CTA performed significantly better than RTA in detecting usability problems. CTA usability problem detection effectiveness was 0.80 vs. 0.62 (p<0.05) respectively, with an average difference of 42% less time spent per subject compared to RTA. In addition, CTA was more thorough in detecting usability problems of a moderate (0.85 vs. 0.7) and severe nature (0.71 vs. 0.57).
In this study, the CTA is more effective in usability-problem detection and provided clarification of intensive care physician query design strategies to inform redesign of the query tool. However, CTA does not outperform RTA. The RTA additionally elucidated unique usability problems and new user requirements. Based on the results of this study, we recommend the use of CTA in formative usability evaluation studies of health information technology. However, we recommend further research on the application of RTA in usability studies with regard to user expertise and experience when focusing on user profile customized (re)design.</description><identifier>ISSN: 1532-0464</identifier><identifier>EISSN: 1532-0480</identifier><identifier>DOI: 10.1016/j.jbi.2015.02.006</identifier><identifier>PMID: 25733166</identifier><language>eng</language><publisher>United States: Elsevier Inc</publisher><subject>Attitude of Health Personnel ; Consumer Behavior - statistics & numerical data ; Data Mining - classification ; Data Mining - methods ; Data Mining - statistics & numerical data ; Design engineering ; Electronic Health Records - statistics & numerical data ; Error detection ; ICU ; Meaningful Use - statistics & numerical data ; Methodologies ; Physicians ; Practice Patterns, Physicians' - classification ; Practice Patterns, Physicians' - statistics & numerical data ; Query processing ; Retrospective Studies ; Software ; Software Validation ; Strategy ; Symbols ; Think Aloud ; Usability ; User requirements ; User–computer interface ; Utilization Review - methods</subject><ispartof>Journal of biomedical informatics, 2015-06, Vol.55, p.1-10</ispartof><rights>2015 Elsevier Inc.</rights><rights>Copyright © 2015 Elsevier Inc. All rights reserved.</rights><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c462t-1883cb58ef3dbc18ce9b013e623480e2a4cd342db06e3d69c3a1ffe33674b1f93</citedby><cites>FETCH-LOGICAL-c462t-1883cb58ef3dbc18ce9b013e623480e2a4cd342db06e3d69c3a1ffe33674b1f93</cites></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://dx.doi.org/10.1016/j.jbi.2015.02.006$$EHTML$$P50$$Gelsevier$$Hfree_for_read</linktohtml><link.rule.ids>314,780,784,3550,27924,27925,45995</link.rule.ids><backlink>$$Uhttps://www.ncbi.nlm.nih.gov/pubmed/25733166$$D View this record in MEDLINE/PubMed$$Hfree_for_read</backlink></links><search><creatorcontrib>Peute, Linda W.P.</creatorcontrib><creatorcontrib>de Keizer, Nicolette F.</creatorcontrib><creatorcontrib>Jaspers, Monique W.M.</creatorcontrib><title>The value of Retrospective and Concurrent Think Aloud in formative usability testing of a physician data query tool</title><title>Journal of biomedical informatics</title><addtitle>J Biomed Inform</addtitle><description>[Display omitted]
•Comparison of Concurrent and Retrospective Think Aloud in usability testing of HIT.•Concurrent and Retrospective Think Aloud differ in usability problem detection scope.•Concurrent Think Aloud has higher effectiveness in terms of system redesign.•Retrospective Think Aloud is promoted for research on expert user customized design.
To compare the performance of the Concurrent (CTA) and Retrospective (RTA) Think Aloud method and to assess their value in a formative usability evaluation of an Intensive Care Registry-physician data query tool designed to support ICU quality improvement processes.
Sixteen representative intensive care physicians participated in the usability evaluation study. Subjects were allocated to either the CTA or RTA method by a matched randomized design. Each subject performed six usability-testing tasks of varying complexity in the query tool in a real-working context. Methods were compared with regard to number and type of problems detected. Verbal protocols of CTA and RTA were analyzed in depth to assess differences in verbal output. Standardized measures were applied to assess thoroughness in usability problem detection weighted per problem severity level and method overall effectiveness in detecting usability problems with regard to the time subjects spent per method.
The usability evaluation of the data query tool revealed a total of 43 unique usability problems that the intensive care physicians encountered. CTA detected unique usability problems with regard to graphics/symbols, navigation issues, error messages, and the organization of information on the query tool’s screens. RTA detected unique issues concerning system match with subjects’ language and applied terminology. The in-depth verbal protocol analysis of CTA provided information on intensive care physicians’ query design strategies. Overall, CTA performed significantly better than RTA in detecting usability problems. CTA usability problem detection effectiveness was 0.80 vs. 0.62 (p<0.05) respectively, with an average difference of 42% less time spent per subject compared to RTA. In addition, CTA was more thorough in detecting usability problems of a moderate (0.85 vs. 0.7) and severe nature (0.71 vs. 0.57).
In this study, the CTA is more effective in usability-problem detection and provided clarification of intensive care physician query design strategies to inform redesign of the query tool. However, CTA does not outperform RTA. The RTA additionally elucidated unique usability problems and new user requirements. Based on the results of this study, we recommend the use of CTA in formative usability evaluation studies of health information technology. However, we recommend further research on the application of RTA in usability studies with regard to user expertise and experience when focusing on user profile customized (re)design.</description><subject>Attitude of Health Personnel</subject><subject>Consumer Behavior - statistics & numerical data</subject><subject>Data Mining - classification</subject><subject>Data Mining - methods</subject><subject>Data Mining - statistics & numerical data</subject><subject>Design engineering</subject><subject>Electronic Health Records - statistics & numerical data</subject><subject>Error detection</subject><subject>ICU</subject><subject>Meaningful Use - statistics & numerical data</subject><subject>Methodologies</subject><subject>Physicians</subject><subject>Practice Patterns, Physicians' - classification</subject><subject>Practice Patterns, Physicians' - statistics & numerical data</subject><subject>Query processing</subject><subject>Retrospective Studies</subject><subject>Software</subject><subject>Software Validation</subject><subject>Strategy</subject><subject>Symbols</subject><subject>Think Aloud</subject><subject>Usability</subject><subject>User requirements</subject><subject>User–computer interface</subject><subject>Utilization Review - methods</subject><issn>1532-0464</issn><issn>1532-0480</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2015</creationdate><recordtype>article</recordtype><sourceid>EIF</sourceid><recordid>eNqNkcFuEzEQhq2Kqi2FB-CCfOSSxWPvenfFqYpKi1QJqQpny2vPNg4bO9jeSHl7HNL2CJw8h2_-8cxHyAdgFTCQnzfVZnAVZ9BUjFeMyTNyBY3gC1Z37M1rLetL8jalDWMATSMvyCVvWiFAyiuSVmukez3NSMNIHzHHkHZostsj1d7SZfBmjhF9pqu18z_pzRRmS52nY4hb_Yebkx7c5PKBZkzZ-adjlKa79SE547SnVmdNf80YCxHC9I6cj3pK-P75vSY_vt6ulveLh-9335Y3DwtTS54X0HXCDE2Ho7CDgc5gPzAQKLko6yHXtbGi5nZgEoWVvREaxhGFkG09wNiLa_LplLuLoUxPWW1dMjhN2mOYk4KW9a2QktX_g0Lds0a2_0Zl15YP9n1XUDihplw1RRzVLrqtjgcFTB0Nqo0qBtXRoGJcFYOl5-Nz_Dxs0b52vCgrwJcTgOV0e4dRJePQG7QuFnHKBveX-N-I_6ws</recordid><startdate>201506</startdate><enddate>201506</enddate><creator>Peute, Linda W.P.</creator><creator>de Keizer, Nicolette F.</creator><creator>Jaspers, Monique W.M.</creator><general>Elsevier Inc</general><scope>6I.</scope><scope>AAFTH</scope><scope>CGR</scope><scope>CUY</scope><scope>CVF</scope><scope>ECM</scope><scope>EIF</scope><scope>NPM</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>7X8</scope><scope>7QO</scope><scope>8FD</scope><scope>FR3</scope><scope>P64</scope><scope>7SC</scope><scope>JQ2</scope><scope>KR7</scope><scope>L7M</scope><scope>L~C</scope><scope>L~D</scope></search><sort><creationdate>201506</creationdate><title>The value of Retrospective and Concurrent Think Aloud in formative usability testing of a physician data query tool</title><author>Peute, Linda W.P. ; de Keizer, Nicolette F. ; Jaspers, Monique W.M.</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c462t-1883cb58ef3dbc18ce9b013e623480e2a4cd342db06e3d69c3a1ffe33674b1f93</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2015</creationdate><topic>Attitude of Health Personnel</topic><topic>Consumer Behavior - statistics & numerical data</topic><topic>Data Mining - classification</topic><topic>Data Mining - methods</topic><topic>Data Mining - statistics & numerical data</topic><topic>Design engineering</topic><topic>Electronic Health Records - statistics & numerical data</topic><topic>Error detection</topic><topic>ICU</topic><topic>Meaningful Use - statistics & numerical data</topic><topic>Methodologies</topic><topic>Physicians</topic><topic>Practice Patterns, Physicians' - classification</topic><topic>Practice Patterns, Physicians' - statistics & numerical data</topic><topic>Query processing</topic><topic>Retrospective Studies</topic><topic>Software</topic><topic>Software Validation</topic><topic>Strategy</topic><topic>Symbols</topic><topic>Think Aloud</topic><topic>Usability</topic><topic>User requirements</topic><topic>User–computer interface</topic><topic>Utilization Review - methods</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Peute, Linda W.P.</creatorcontrib><creatorcontrib>de Keizer, Nicolette F.</creatorcontrib><creatorcontrib>Jaspers, Monique W.M.</creatorcontrib><collection>ScienceDirect Open Access Titles</collection><collection>Elsevier:ScienceDirect:Open Access</collection><collection>Medline</collection><collection>MEDLINE</collection><collection>MEDLINE (Ovid)</collection><collection>MEDLINE</collection><collection>MEDLINE</collection><collection>PubMed</collection><collection>CrossRef</collection><collection>MEDLINE - Academic</collection><collection>Biotechnology Research Abstracts</collection><collection>Technology Research Database</collection><collection>Engineering Research Database</collection><collection>Biotechnology and BioEngineering Abstracts</collection><collection>Computer and Information Systems Abstracts</collection><collection>ProQuest Computer Science Collection</collection><collection>Civil Engineering Abstracts</collection><collection>Advanced Technologies Database with Aerospace</collection><collection>Computer and Information Systems Abstracts Academic</collection><collection>Computer and Information Systems Abstracts Professional</collection><jtitle>Journal of biomedical informatics</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Peute, Linda W.P.</au><au>de Keizer, Nicolette F.</au><au>Jaspers, Monique W.M.</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>The value of Retrospective and Concurrent Think Aloud in formative usability testing of a physician data query tool</atitle><jtitle>Journal of biomedical informatics</jtitle><addtitle>J Biomed Inform</addtitle><date>2015-06</date><risdate>2015</risdate><volume>55</volume><spage>1</spage><epage>10</epage><pages>1-10</pages><issn>1532-0464</issn><eissn>1532-0480</eissn><abstract>[Display omitted]
•Comparison of Concurrent and Retrospective Think Aloud in usability testing of HIT.•Concurrent and Retrospective Think Aloud differ in usability problem detection scope.•Concurrent Think Aloud has higher effectiveness in terms of system redesign.•Retrospective Think Aloud is promoted for research on expert user customized design.
To compare the performance of the Concurrent (CTA) and Retrospective (RTA) Think Aloud method and to assess their value in a formative usability evaluation of an Intensive Care Registry-physician data query tool designed to support ICU quality improvement processes.
Sixteen representative intensive care physicians participated in the usability evaluation study. Subjects were allocated to either the CTA or RTA method by a matched randomized design. Each subject performed six usability-testing tasks of varying complexity in the query tool in a real-working context. Methods were compared with regard to number and type of problems detected. Verbal protocols of CTA and RTA were analyzed in depth to assess differences in verbal output. Standardized measures were applied to assess thoroughness in usability problem detection weighted per problem severity level and method overall effectiveness in detecting usability problems with regard to the time subjects spent per method.
The usability evaluation of the data query tool revealed a total of 43 unique usability problems that the intensive care physicians encountered. CTA detected unique usability problems with regard to graphics/symbols, navigation issues, error messages, and the organization of information on the query tool’s screens. RTA detected unique issues concerning system match with subjects’ language and applied terminology. The in-depth verbal protocol analysis of CTA provided information on intensive care physicians’ query design strategies. Overall, CTA performed significantly better than RTA in detecting usability problems. CTA usability problem detection effectiveness was 0.80 vs. 0.62 (p<0.05) respectively, with an average difference of 42% less time spent per subject compared to RTA. In addition, CTA was more thorough in detecting usability problems of a moderate (0.85 vs. 0.7) and severe nature (0.71 vs. 0.57).
In this study, the CTA is more effective in usability-problem detection and provided clarification of intensive care physician query design strategies to inform redesign of the query tool. However, CTA does not outperform RTA. The RTA additionally elucidated unique usability problems and new user requirements. Based on the results of this study, we recommend the use of CTA in formative usability evaluation studies of health information technology. However, we recommend further research on the application of RTA in usability studies with regard to user expertise and experience when focusing on user profile customized (re)design.</abstract><cop>United States</cop><pub>Elsevier Inc</pub><pmid>25733166</pmid><doi>10.1016/j.jbi.2015.02.006</doi><tpages>10</tpages><oa>free_for_read</oa></addata></record> |
| fulltext | fulltext |
| identifier | ISSN: 1532-0464 |
| ispartof | Journal of biomedical informatics, 2015-06, Vol.55, p.1-10 |
| issn | 1532-0464 1532-0480 |
| language | eng |
| recordid | cdi_proquest_miscellaneous_1709736604 |
| source | MEDLINE; Elektronische Zeitschriftenbibliothek - Frei zugängliche E-Journals; Access via ScienceDirect (Elsevier) |
| subjects | Attitude of Health Personnel Consumer Behavior - statistics & numerical data Data Mining - classification Data Mining - methods Data Mining - statistics & numerical data Design engineering Electronic Health Records - statistics & numerical data Error detection ICU Meaningful Use - statistics & numerical data Methodologies Physicians Practice Patterns, Physicians' - classification Practice Patterns, Physicians' - statistics & numerical data Query processing Retrospective Studies Software Software Validation Strategy Symbols Think Aloud Usability User requirements User–computer interface Utilization Review - methods |
| title | The value of Retrospective and Concurrent Think Aloud in formative usability testing of a physician data query tool |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-23T10%3A39%3A28IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_cross&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=The%20value%20of%20Retrospective%20and%20Concurrent%20Think%20Aloud%20in%20formative%20usability%20testing%20of%20a%20physician%20data%20query%20tool&rft.jtitle=Journal%20of%20biomedical%20informatics&rft.au=Peute,%20Linda%20W.P.&rft.date=2015-06&rft.volume=55&rft.spage=1&rft.epage=10&rft.pages=1-10&rft.issn=1532-0464&rft.eissn=1532-0480&rft_id=info:doi/10.1016/j.jbi.2015.02.006&rft_dat=%3Cproquest_cross%3E1701490567%3C/proquest_cross%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=1687348998&rft_id=info:pmid/25733166&rft_els_id=S1532046415000386&rfr_iscdi=true |