The multiscale top-hat tensor enables specific enhancement of curvilinear structures in 2D and 3D images

•Improved method that combined the power of morphological operations with local tensor.•A multiscale process to enhance the curvilinear structures in different directions.•Implemented in both 2D and 3D.•Comparable to the state-of-the-art for the various challenging retina image datasets.•Show good results in term of specificity compared with the segmentation methods. Quantification and modelling of curvilinear structures in 2D and 3D images is a common challenge in a wide range of biomedical applications. Image enhancement is a crucial pre-processing step for curvilinear structure quantification. Many of the existing state-of-the-art enhancement approaches still suffer from contrast variations and noise. In this paper, we propose to address such problems via the use of a multiscale image processing approach, called Multiscale Top-Hat Tensor (MTHT). MTHT produces a better quality enhancement of curvilinear structures in low contrast and noisy images compared with other approaches in a range of 2D and 3D biomedical images. The proposed approach combines multiscale morphological filtering with a local tensor representation of curvilinear structure. The MTHT approach is validated on 2D and 3D synthetic and real images, and is also compared to the state-of-the-art curvilinear structure enhancement approaches. The obtained results demonstrate that the proposed approach provides high-quality curvilinear structure enhancement, allowing high accuracy segmentation and quantification in a wide range of 2D and 3D image datasets.