Interpretable ensembles of hyper-rectangles as base models

A new extremely simple ensemble-based model with the uniformly generated axis-parallel hyper-rectangles as base models (HRBM) is proposed. Two types of HRBMs are studied: closed rectangles and corners. The main idea behind HRBM is to consider training examples inside and outside each rectangle. It is proposed to incorporate HRBMs into the gradient boosting machine (GBM) to construct effective ensemble-based models and to avoid overfitting. A simple method for calculating optimal regularization parameters of the model, which can be modified in the explicit way at each iteration of GBM, is considered. Moreover, a new regularization called the “step height penalty” is studied in addition to the L 1 and L 2 regularizations. An extremely simple approach to the proposed ensemble-based model prediction interpretation by using the well-known method SHAP is proposed. It is shown that GBM with HRBM can be regarded as a model extending a set of interpretable models for explaining black-box models. Numerical experiments with real datasets illustrate the proposed GBM with HRBMs for regression and classification problems. The best p -values in the t -test comparing the proposed model with the well-known ensemble-based regression and classification models are 0.004 and 0.0031, respectively. Experiments also illustrate computational efficiency of the proposed SHAP modifications. The code of proposed algorithms implementing GBM with HRBM is publicly available.