Order selection for regression-based hidden Markov model

Hidden Markov models (HMMs) describe the relationship between two stochastic processes: an observed process and an unobservable finite-state transition process. Owing to their modeling dynamic heterogeneity, HMMs are widely used to analyze heterogeneous longitudinal data. Traditional HMMs frequently assume that the number of hidden states (i.e., the order of HMM) is a constant and should be specified prior to analysis. This assumption is unrealistic and restrictive in many applications. In this study, we consider regression-based hidden Markov model (RHMM) while allowing the number of hidden states to be unknown and determined by the data. We propose a novel likelihood-based double penalized method, along with an efficient expectation-conditional maximization with iterative thresholding-based descent (ECM–ITD) algorithm, to perform order selection in the context of RHMM. An extended Group-Sort-Fuse procedure is proposed to rank the regression coefficients and impose penalties on the discrepancy of adjacent coefficients. The order selection consistency and convergence of the ECM–ITD algorithm are established under mild conditions. Simulation studies are conducted to evaluate the empirical performance of the proposed method. An application of the proposed methodology to a real-life study on Alzheimer’s disease is presented.