LBP operators on curvelet coefficients as an algorithm to describe texture in breast cancer tissues

•We present a method based on curvelet transform, LBP, ANOVA and PL classifier.•We validate the proposed approach considering the metrics accuracy and AUC.•The features was evaluated by applying the DT, RaF, SVM and PL classifiers.•The proposed approach achieved AC values among 91% and 100%.•The method was tested on the datasets: DDSM, BCDR-FMR, BCDR-DMR and UCSB-BB. In computer-aided diagnosis one of the crucial steps to classify suspicious lesions is the extraction of features. Texture analysis methods have been used in the analysis and interpretation of medical images. In this work we present a method based on the association among curvelet transform, local binary patterns, feature selection by statistical analysis and distinct classification methods, in order to support the development of computer aided diagnosis system. The similar features were removed by the statistical analysis of variance (ANOVA). The understanding of the features was evaluated by applying the decision tree, random forest, support vector machine and polynomial (PL) classifiers, considering the metrics accuracy (AC) and area under the ROC curve (AUC): the rates were calculated on images of breast tissues with different physical properties (commonly observed in clinical practice). The datasets were the Digital Database for Screening Mammography, Breast Cancer Digital Repository and UCSB biosegmentation benchmark. The investigated groups were normal-abnormal and benign-malignant. The association of curvelet transform, local binary pattern and ANOVA with the PL classifier achieved higher AUC and AC values for all cases: the obtained rates were among 91% and 100%. These results are relevant, specially when we consider the difficulties of clinical practice in distinguishing the studied groups. The proposed association is useful as an automated protocol for the diagnosis of breast tissues and may contribute to the diagnosis of breast tissues (mammographic and histopathological images).