MEASURING THE VALIDITY AND RELIABILITY OF VALUE ASSESSMENT FRAMEWORKS FOR CANCER DRUGS: AN EVALUATION METHOD

OBJECTIVES: We aimed to develop a methodology for evaluating convergent validity and inter-rater reliability of value assessment frameworks. METHODS: Framework convergent validity, defined as the correlation among drug rankings across frameworks, can be assessed using Kendall's W coefficient. Framework reliability is evaluated using intraclass correlation coefficients (ICC), which measure the stability of outcomes across users. Drugs can be assessed by independent physician and non-physician evaluators, who can use published drug trial data and instructions provided by framework developers to assign each drug a numeric or letter score. Mean scores for drugs within pre-defined categories (e.g., condition; indication) are rank-ordered to estimate Kendall's W. Multiple scores for the same drug are compared using ICC. To evaluate stability of results, W and ICC are assessed with varying numbers of evaluators and frameworks. The method was applied here by 8 evaluators, who assessed 15 oncology drugs and completed a survey on their experiences. RESULTS: Excluding review of drug trial data, each assessment took on average 25 minutes for ASCO, 21 for ICER, 14 for ESMO, and 8 for NCCN. Mean time to review each drug's data was 20 minutes. Kendall's W was 0.560 (p=0.010), 0.562 (p=0.010), and 0.920 (p