MEASURING THE VALIDITY AND RELIABILITY OF VALUE ASSESSMENT FRAMEWORKS FOR CANCER DRUGS: AN EVALUATION METHOD
OBJECTIVES: We aimed to develop a methodology for evaluating convergent validity and inter-rater reliability of value assessment frameworks. METHODS: Framework convergent validity, defined as the correlation among drug rankings across frameworks, can be assessed using Kendall's W coefficient. F...
Gespeichert in:
| Veröffentlicht in: | Value in health 2017-05, Vol.20 (5), p.A341 |
|---|---|
| Hauptverfasser: | , , , , , , , , , , , , |
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | |
|---|---|
| container_issue | 5 |
| container_start_page | A341 |
| container_title | Value in health |
| container_volume | 20 |
| creator | Bentley, TG Cohen, JT Elkin, EB Huynh, J Mukherjea, A Neville, TH Mei, MG Copher, R Knoth, RL Popescu, I Lee, J Zambrano, JM Broder, MS |
| description | OBJECTIVES: We aimed to develop a methodology for evaluating convergent validity and inter-rater reliability of value assessment frameworks. METHODS: Framework convergent validity, defined as the correlation among drug rankings across frameworks, can be assessed using Kendall's W coefficient. Framework reliability is evaluated using intraclass correlation coefficients (ICC), which measure the stability of outcomes across users. Drugs can be assessed by independent physician and non-physician evaluators, who can use published drug trial data and instructions provided by framework developers to assign each drug a numeric or letter score. Mean scores for drugs within pre-defined categories (e.g., condition; indication) are rank-ordered to estimate Kendall's W. Multiple scores for the same drug are compared using ICC. To evaluate stability of results, W and ICC are assessed with varying numbers of evaluators and frameworks. The method was applied here by 8 evaluators, who assessed 15 oncology drugs and completed a survey on their experiences. RESULTS: Excluding review of drug trial data, each assessment took on average 25 minutes for ASCO, 21 for ICER, 14 for ESMO, and 8 for NCCN. Mean time to review each drug's data was 20 minutes. Kendall's W was 0.560 (p=0.010), 0.562 (p=0.010), and 0.920 (p |
| doi_str_mv | 10.1016/j.jval.2017.05.005 |
| format | Article |
| fullrecord | <record><control><sourceid>proquest</sourceid><recordid>TN_cdi_proquest_journals_2097657307</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2097657307</sourcerecordid><originalsourceid>FETCH-proquest_journals_20976573073</originalsourceid><addsrcrecordid>eNqNjM1OwzAQhC0EEuXnBTitxDlmHdcx5WaSTWORxJLtgDhVPZRDFFFoKM_fROIBOM2Mvplh7E4gFyiyh573v9uBpyg0R8UR1RlbCJUuk6WW8nzyuHpMJAp1ya7GsUfETKZqwYaGTOi8bdcQK4JXU9vCxncwbQGeamuebT1nV86sIzAhUAgNtRFKbxp6c_4lQOk85KbNyUPhu3V4mg6A5oWJ1rXQUKxcccMuPrbDuLv902t2X1LMq-TrsP8-7safTb8_Hj4ntElxpTOlJWr5v9YJzIBGKg</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2097657307</pqid></control><display><type>article</type><title>MEASURING THE VALIDITY AND RELIABILITY OF VALUE ASSESSMENT FRAMEWORKS FOR CANCER DRUGS: AN EVALUATION METHOD</title><source>Applied Social Sciences Index & Abstracts (ASSIA)</source><source>Access via ScienceDirect (Elsevier)</source><source>EZB-FREE-00999 freely available EZB journals</source><creator>Bentley, TG ; Cohen, JT ; Elkin, EB ; Huynh, J ; Mukherjea, A ; Neville, TH ; Mei, MG ; Copher, R ; Knoth, RL ; Popescu, I ; Lee, J ; Zambrano, JM ; Broder, MS</creator><creatorcontrib>Bentley, TG ; Cohen, JT ; Elkin, EB ; Huynh, J ; Mukherjea, A ; Neville, TH ; Mei, MG ; Copher, R ; Knoth, RL ; Popescu, I ; Lee, J ; Zambrano, JM ; Broder, MS</creatorcontrib><description>OBJECTIVES: We aimed to develop a methodology for evaluating convergent validity and inter-rater reliability of value assessment frameworks. METHODS: Framework convergent validity, defined as the correlation among drug rankings across frameworks, can be assessed using Kendall's W coefficient. Framework reliability is evaluated using intraclass correlation coefficients (ICC), which measure the stability of outcomes across users. Drugs can be assessed by independent physician and non-physician evaluators, who can use published drug trial data and instructions provided by framework developers to assign each drug a numeric or letter score. Mean scores for drugs within pre-defined categories (e.g., condition; indication) are rank-ordered to estimate Kendall's W. Multiple scores for the same drug are compared using ICC. To evaluate stability of results, W and ICC are assessed with varying numbers of evaluators and frameworks. The method was applied here by 8 evaluators, who assessed 15 oncology drugs and completed a survey on their experiences. RESULTS: Excluding review of drug trial data, each assessment took on average 25 minutes for ASCO, 21 for ICER, 14 for ESMO, and 8 for NCCN. Mean time to review each drug's data was 20 minutes. Kendall's W was 0.560 (p=0.010), 0.562 (p=0.010), and 0.920 (p<0.001) for drugs assessed in breast, lung, and prostate cancers, respectively. ICC (95% CI) for ASCO, ESMO, ICER, and NCCN were: 0.800 (0.660-0.913), 0.818 (0.686-0.921), 0.652 (0.466-0.834), and 0.153 (0.0045-0.371), respectively. Evaluators considered framework usability reasonable with this method. CONCLUSIONS: This novel method allows quantitative analyses of value assessment frameworks' convergent validity and inter-rater reliability. Although the approach can be used to determine the reproducibility of value assessments produced by these frameworks, the true test of their contribution to value-based decision-making will be how they influence decisions made by clinicians and patients when used in clinical practice settings.</description><identifier>ISSN: 1098-3015</identifier><identifier>EISSN: 1524-4733</identifier><identifier>DOI: 10.1016/j.jval.2017.05.005</identifier><language>eng</language><publisher>Lawrenceville: Elsevier Science Ltd</publisher><subject>Averages ; Breast cancer ; Cancer ; Cancer therapies ; Clinical medicine ; Convergent validity ; Decision making ; Drug delivery systems ; Drug development ; Drug screening ; Evaluation ; Lung cancer ; Oncology ; Prostate ; Prostate cancer ; Reliability ; Reproducibility ; Value</subject><ispartof>Value in health, 2017-05, Vol.20 (5), p.A341</ispartof><rights>Copyright Elsevier Science Ltd. May 2017</rights><lds50>peer_reviewed</lds50><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>314,780,784,27924,27925,30999</link.rule.ids></links><search><creatorcontrib>Bentley, TG</creatorcontrib><creatorcontrib>Cohen, JT</creatorcontrib><creatorcontrib>Elkin, EB</creatorcontrib><creatorcontrib>Huynh, J</creatorcontrib><creatorcontrib>Mukherjea, A</creatorcontrib><creatorcontrib>Neville, TH</creatorcontrib><creatorcontrib>Mei, MG</creatorcontrib><creatorcontrib>Copher, R</creatorcontrib><creatorcontrib>Knoth, RL</creatorcontrib><creatorcontrib>Popescu, I</creatorcontrib><creatorcontrib>Lee, J</creatorcontrib><creatorcontrib>Zambrano, JM</creatorcontrib><creatorcontrib>Broder, MS</creatorcontrib><title>MEASURING THE VALIDITY AND RELIABILITY OF VALUE ASSESSMENT FRAMEWORKS FOR CANCER DRUGS: AN EVALUATION METHOD</title><title>Value in health</title><description>OBJECTIVES: We aimed to develop a methodology for evaluating convergent validity and inter-rater reliability of value assessment frameworks. METHODS: Framework convergent validity, defined as the correlation among drug rankings across frameworks, can be assessed using Kendall's W coefficient. Framework reliability is evaluated using intraclass correlation coefficients (ICC), which measure the stability of outcomes across users. Drugs can be assessed by independent physician and non-physician evaluators, who can use published drug trial data and instructions provided by framework developers to assign each drug a numeric or letter score. Mean scores for drugs within pre-defined categories (e.g., condition; indication) are rank-ordered to estimate Kendall's W. Multiple scores for the same drug are compared using ICC. To evaluate stability of results, W and ICC are assessed with varying numbers of evaluators and frameworks. The method was applied here by 8 evaluators, who assessed 15 oncology drugs and completed a survey on their experiences. RESULTS: Excluding review of drug trial data, each assessment took on average 25 minutes for ASCO, 21 for ICER, 14 for ESMO, and 8 for NCCN. Mean time to review each drug's data was 20 minutes. Kendall's W was 0.560 (p=0.010), 0.562 (p=0.010), and 0.920 (p<0.001) for drugs assessed in breast, lung, and prostate cancers, respectively. ICC (95% CI) for ASCO, ESMO, ICER, and NCCN were: 0.800 (0.660-0.913), 0.818 (0.686-0.921), 0.652 (0.466-0.834), and 0.153 (0.0045-0.371), respectively. Evaluators considered framework usability reasonable with this method. CONCLUSIONS: This novel method allows quantitative analyses of value assessment frameworks' convergent validity and inter-rater reliability. Although the approach can be used to determine the reproducibility of value assessments produced by these frameworks, the true test of their contribution to value-based decision-making will be how they influence decisions made by clinicians and patients when used in clinical practice settings.</description><subject>Averages</subject><subject>Breast cancer</subject><subject>Cancer</subject><subject>Cancer therapies</subject><subject>Clinical medicine</subject><subject>Convergent validity</subject><subject>Decision making</subject><subject>Drug delivery systems</subject><subject>Drug development</subject><subject>Drug screening</subject><subject>Evaluation</subject><subject>Lung cancer</subject><subject>Oncology</subject><subject>Prostate</subject><subject>Prostate cancer</subject><subject>Reliability</subject><subject>Reproducibility</subject><subject>Value</subject><issn>1098-3015</issn><issn>1524-4733</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2017</creationdate><recordtype>article</recordtype><sourceid>7QJ</sourceid><recordid>eNqNjM1OwzAQhC0EEuXnBTitxDlmHdcx5WaSTWORxJLtgDhVPZRDFFFoKM_fROIBOM2Mvplh7E4gFyiyh573v9uBpyg0R8UR1RlbCJUuk6WW8nzyuHpMJAp1ya7GsUfETKZqwYaGTOi8bdcQK4JXU9vCxncwbQGeamuebT1nV86sIzAhUAgNtRFKbxp6c_4lQOk85KbNyUPhu3V4mg6A5oWJ1rXQUKxcccMuPrbDuLv902t2X1LMq-TrsP8-7safTb8_Hj4ntElxpTOlJWr5v9YJzIBGKg</recordid><startdate>20170501</startdate><enddate>20170501</enddate><creator>Bentley, TG</creator><creator>Cohen, JT</creator><creator>Elkin, EB</creator><creator>Huynh, J</creator><creator>Mukherjea, A</creator><creator>Neville, TH</creator><creator>Mei, MG</creator><creator>Copher, R</creator><creator>Knoth, RL</creator><creator>Popescu, I</creator><creator>Lee, J</creator><creator>Zambrano, JM</creator><creator>Broder, MS</creator><general>Elsevier Science Ltd</general><scope>7QJ</scope></search><sort><creationdate>20170501</creationdate><title>MEASURING THE VALIDITY AND RELIABILITY OF VALUE ASSESSMENT FRAMEWORKS FOR CANCER DRUGS: AN EVALUATION METHOD</title><author>Bentley, TG ; Cohen, JT ; Elkin, EB ; Huynh, J ; Mukherjea, A ; Neville, TH ; Mei, MG ; Copher, R ; Knoth, RL ; Popescu, I ; Lee, J ; Zambrano, JM ; Broder, MS</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-proquest_journals_20976573073</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2017</creationdate><topic>Averages</topic><topic>Breast cancer</topic><topic>Cancer</topic><topic>Cancer therapies</topic><topic>Clinical medicine</topic><topic>Convergent validity</topic><topic>Decision making</topic><topic>Drug delivery systems</topic><topic>Drug development</topic><topic>Drug screening</topic><topic>Evaluation</topic><topic>Lung cancer</topic><topic>Oncology</topic><topic>Prostate</topic><topic>Prostate cancer</topic><topic>Reliability</topic><topic>Reproducibility</topic><topic>Value</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Bentley, TG</creatorcontrib><creatorcontrib>Cohen, JT</creatorcontrib><creatorcontrib>Elkin, EB</creatorcontrib><creatorcontrib>Huynh, J</creatorcontrib><creatorcontrib>Mukherjea, A</creatorcontrib><creatorcontrib>Neville, TH</creatorcontrib><creatorcontrib>Mei, MG</creatorcontrib><creatorcontrib>Copher, R</creatorcontrib><creatorcontrib>Knoth, RL</creatorcontrib><creatorcontrib>Popescu, I</creatorcontrib><creatorcontrib>Lee, J</creatorcontrib><creatorcontrib>Zambrano, JM</creatorcontrib><creatorcontrib>Broder, MS</creatorcontrib><collection>Applied Social Sciences Index & Abstracts (ASSIA)</collection><jtitle>Value in health</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Bentley, TG</au><au>Cohen, JT</au><au>Elkin, EB</au><au>Huynh, J</au><au>Mukherjea, A</au><au>Neville, TH</au><au>Mei, MG</au><au>Copher, R</au><au>Knoth, RL</au><au>Popescu, I</au><au>Lee, J</au><au>Zambrano, JM</au><au>Broder, MS</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>MEASURING THE VALIDITY AND RELIABILITY OF VALUE ASSESSMENT FRAMEWORKS FOR CANCER DRUGS: AN EVALUATION METHOD</atitle><jtitle>Value in health</jtitle><date>2017-05-01</date><risdate>2017</risdate><volume>20</volume><issue>5</issue><spage>A341</spage><pages>A341-</pages><issn>1098-3015</issn><eissn>1524-4733</eissn><abstract>OBJECTIVES: We aimed to develop a methodology for evaluating convergent validity and inter-rater reliability of value assessment frameworks. METHODS: Framework convergent validity, defined as the correlation among drug rankings across frameworks, can be assessed using Kendall's W coefficient. Framework reliability is evaluated using intraclass correlation coefficients (ICC), which measure the stability of outcomes across users. Drugs can be assessed by independent physician and non-physician evaluators, who can use published drug trial data and instructions provided by framework developers to assign each drug a numeric or letter score. Mean scores for drugs within pre-defined categories (e.g., condition; indication) are rank-ordered to estimate Kendall's W. Multiple scores for the same drug are compared using ICC. To evaluate stability of results, W and ICC are assessed with varying numbers of evaluators and frameworks. The method was applied here by 8 evaluators, who assessed 15 oncology drugs and completed a survey on their experiences. RESULTS: Excluding review of drug trial data, each assessment took on average 25 minutes for ASCO, 21 for ICER, 14 for ESMO, and 8 for NCCN. Mean time to review each drug's data was 20 minutes. Kendall's W was 0.560 (p=0.010), 0.562 (p=0.010), and 0.920 (p<0.001) for drugs assessed in breast, lung, and prostate cancers, respectively. ICC (95% CI) for ASCO, ESMO, ICER, and NCCN were: 0.800 (0.660-0.913), 0.818 (0.686-0.921), 0.652 (0.466-0.834), and 0.153 (0.0045-0.371), respectively. Evaluators considered framework usability reasonable with this method. CONCLUSIONS: This novel method allows quantitative analyses of value assessment frameworks' convergent validity and inter-rater reliability. Although the approach can be used to determine the reproducibility of value assessments produced by these frameworks, the true test of their contribution to value-based decision-making will be how they influence decisions made by clinicians and patients when used in clinical practice settings.</abstract><cop>Lawrenceville</cop><pub>Elsevier Science Ltd</pub><doi>10.1016/j.jval.2017.05.005</doi></addata></record> |
| fulltext | fulltext |
| identifier | ISSN: 1098-3015 |
| ispartof | Value in health, 2017-05, Vol.20 (5), p.A341 |
| issn | 1098-3015 1524-4733 |
| language | eng |
| recordid | cdi_proquest_journals_2097657307 |
| source | Applied Social Sciences Index & Abstracts (ASSIA); Access via ScienceDirect (Elsevier); EZB-FREE-00999 freely available EZB journals |
| subjects | Averages Breast cancer Cancer Cancer therapies Clinical medicine Convergent validity Decision making Drug delivery systems Drug development Drug screening Evaluation Lung cancer Oncology Prostate Prostate cancer Reliability Reproducibility Value |
| title | MEASURING THE VALIDITY AND RELIABILITY OF VALUE ASSESSMENT FRAMEWORKS FOR CANCER DRUGS: AN EVALUATION METHOD |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-23T19%3A50%3A47IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=MEASURING%20THE%20VALIDITY%20AND%20RELIABILITY%20OF%20VALUE%20ASSESSMENT%20FRAMEWORKS%20FOR%20CANCER%20DRUGS:%20AN%20EVALUATION%20METHOD&rft.jtitle=Value%20in%20health&rft.au=Bentley,%20TG&rft.date=2017-05-01&rft.volume=20&rft.issue=5&rft.spage=A341&rft.pages=A341-&rft.issn=1098-3015&rft.eissn=1524-4733&rft_id=info:doi/10.1016/j.jval.2017.05.005&rft_dat=%3Cproquest%3E2097657307%3C/proquest%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2097657307&rft_id=info:pmid/&rfr_iscdi=true |