Cancer detection with various classification models: A comprehensive feature analysis using HMM to extract a nucleotide pattern

This work presents a novel feature extraction method for identifying complex patterns in genomic sequences by employing the Hidden Markov Model (HMM). In this study, we use HMM to identify gene nucleotide patterns that are specific to malignant and non-malignant cells. Crucial genetic components DNA and RNA are involved in many biological processes that impact both healthy and malignant cells. Early patient identification is essential to successful cancer diagnosis and therapy. Varying nucleotide patterns indicate different cellular responses, which are important to understanding the molecular causes of cancer and associated disorders. We present a detailed study of nucleotide patterns in whole raw nucleotide sequences with variations in both protein sequence (CDS) and non-protein sequence (NCDS) in both malignant and non-malignant cells. Nucleotide prediction has been achieved while computational expenses are reduced by using the proposed HMM for feature extraction and selection. The classification models implemented in this work for cancer detection are Gradient-Boosted Decision Trees (GBDT), Random Forests (RF), Decision Trees (DT), and Support Vector Machines (SVM) with kernels. The suggested classification model's accuracy and 10-fold cross-validation have been validated via comprehensive case studies. The results reveal that DT and ensemble learning techniques significantly differentiate between malignant and non-malignant DNA sequences. SVM with suitable kernels improves cancer detection accuracy significantly. Combining feature reduction approaches with nucleotide pattern classifiers based on Hidden Markov models improves performance and ensures reliable cancer detection. [Display omitted] •The HMM is used for a novel feature extraction approach to find complicated patterns in genomic sequences.•Comprehensive feature extraction of nucleotide patterns in whole sequences (CDS/NCDS) for malignant and non-malignant cells.•Here DT and ensemble learning algorithms effectively distinguish malignant and non-malignant DNA sequences.•The use of SVM with appropriate kernels significantly improves accuracy in cancer diagnosis.