VerSe: A Vertebrae labelling and segmentation benchmark for multi-detector CT images

•A total of 374 multi-detector CT scans are made available to the research community, the biggest such dataset on spine until date (VerSe’19: https://osf.io/nqjyw/; VerSe’20: https://osf.io/t98fz/).•The VerSe benchmark includes annotations for two fundamental processing tasks, namely vertebrae labelling and segmentation.•Twenty-six fully-automated algorithms (eleven for VerSe’19, thirteen for VerSe’20, and one baseline) are benchmarked on this dataset, with the top performing algorithm achieving a mean vertebrae identification rate of 96.6% and a Dice coefficient of 91.7% in VerSe’20.•Further insights into these algorithms are provided by examining them at various levels of granularity ranging from dataset-level experiments to vertebrae-level performances to a field-of-view-related analysis. [Display omitted] Vertebral labelling and segmentation are two fundamental tasks in an automated spine processing pipeline. Reliable and accurate processing of spine images is expected to benefit clinical decision support systems for diagnosis, surgery planning, and population-based analysis of spine and bone health. However, designing automated algorithms for spine processing is challenging predominantly due to considerable variations in anatomy and acquisition protocols and due to a severe shortage of publicly available data. Addressing these limitations, the Large Scale Vertebrae Segmentation Challenge (VerSe) was organised in conjunction with the International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) in 2019 and 2020, with a call for algorithms tackling the labelling and segmentation of vertebrae. Two datasets containing a total of 374 multi-detector CT scans from 355 patients were prepared and 4505 vertebrae have individually been annotated at voxel level by a human-machine hybrid algorithm (https://osf.io/nqjyw/, https://osf.io/t98fz/). A total of 25 algorithms were benchmarked on these datasets. In this work, we present the results of this evaluation and further investigate the performance variation at the vertebra level, scan level, and different fields of view. We also evaluate the generalisability of the approaches to an implicit domain shift in data by evaluating the top-performing algorithms of one challenge iteration on data from the other iteration. The principal takeaway from VerSe: the performance of an algorithm in labelling and segmenting a spine scan hinges on its ability to correctly identify vertebrae in cas