Holistic multitask regression network for multiapplication shape regression segmentation

•Achieved multiapplication shape regression segmentation in a single framework.•Formulated segmentation as a multitask regression task; models task correlations.•Direct coordinate estimation of an organ's shape contour, jointly captures shape.•A multitask regression deep learning framework with multiscale+fused representation. [Display omitted] A holistic multitask regression approach was implemented to tackle the limitations of clinical image analysis. Standard practice requires identifying multiple anatomic structures in multiple planes from multiple anatomic regions using multiple modalities. The proposed novel holistic multitask regression network (HMR-Net) formulates organ segmentation as a multitask learning problem. Multitask learning leverages the strength of joint task problem solving from capturing task correlations. HMR-Net performs multitask regression by estimating an organ’s class, regional location, and precise contour coordinates. The estimation of each coordinate point also corresponds to another regression task. HMR-Net leverages hierarchical multiscale and fused organ features to handle nonlinear relationships between image appearance and distinct organ properties. Simultaneously, holistic shape information is captured by encoding coordinate correlations. The multitask pipeline enables the capturing of holistic organ information (e.g. class, location, shape) to perform shape regression for medical image segmentation. HMR-Net was validated on eight representative datasets obtained from a total of 222 subjects. A mean average precision and dice score reaching up to 0.81 and 0.93, respectively, was achieved on the representative multiapplication database. The generalized model demonstrates comparable or superior performance compared to state-of-the-art algorithms. The high-performance accuracy demonstrates our model as an effective general framework to perform organ shape regression in multiple applications. This method was proven to provide high-contrast sensitivity to delineate even the smallest and oddly shaped organs. HMR-Net’s flexible framework holds great potential in providing a fully automatic preliminary analysis for multiple types of medical images.