Can measuring hippocampal atrophy with a fully automatic method be substantially less noisy than manual segmentation over both 1 and 3 years?

•Robust assessment of segmentation noise of atrophy measuring methods in Alzheimer's.•The noise of all methods violated the Gaussianity assumption of common statistics.•Using robust statistics FreeSurfer 5.3.0 had the same noise as manual segmentation.•MAPS-HBSI and FreeSurfer 6.0.0 were substantially less noisy than manual.•MAPS-HBSI and FreeSurfer 6.0.0 required less than two thirds of the subjects of manual. To quantify the “segmentation noise” of several widely used fully automatic methods for measuring longitudinal hippocampal atrophy in Alzheimer's disease and compare the results to the segmentation noise of manual segmentation over both 1 and 3 years. The segmentation noise of 5 longitudinal hippocampal atrophy measurement methods was quantified, including checking its Gaussianity, using 264 subjects from the ADNI1 back-to-back (BTB) data set over both 1 year and 3 year intervals. The segmentation methods were FreeSurfer 5.3.0 both cross sectional and longitudinal, FreeSurfer 6.0.0 longitudinal, MAPS-HBSI and FSL/FIRST 5.0.8. The BTB manual segmentation of 75 ADNI subjects from a previous study provided the manual distributions for comparison. All methods, including the manual segmentation, violated the Gaussianity assumption. Two methods, FreeSurfer 6.0.0 and MAPS-HBSI, had a segmentation noise substantially less than a surrogate for manual segmentation. FreeSurfer 5.3.0 longitudinal was confirmed as a surrogate for manual segmentation. The violation of the Gaussian assumption by the segmentation methods assessed, including manual, suggests results of previous studies that assumed Gaussian statistics without confirmation may need review. Fully automatic FreeSurfer 6.0.0 and MAPS-HBSI both have lower segmentation noise than manual requiring less than two thirds of the subjects to detect the same treatment effect.