Identifying optimally cost-effective dynamic treatment regimes with a Q-learning approach

Health policy decisions regarding patient treatment strategies require consideration of both treatment effectiveness and cost. We propose a two-step approach for identifying an optimally cost-effective and interpretable dynamic treatment regime. First, we develop a combined Q-learning and policy-search approach to estimate optimal list-based regimes under a constraint on expected treatment costs. Second, we propose an iterative procedure to select an optimally cost-effective regime from a set of candidate regimes corresponding to different cost constraints. Our approach can estimate optimal regimes in the presence of time-varying confounding, censoring, and correlated outcomes. Through simulation studies, we examine the operating characteristics of our approach under flexible modelling approaches. We also apply our methodology to identify optimally cost-effective treatment strategies for assigning adjuvant therapies to endometrial cancer patients.