Growth defects in GeSn/Ge/Si(001) epitaxial layers grown by hot wire chemical vapor deposition of Ge with co-evaporation of Sn

•GeSn/Ge/Si(001) epitaxial layers were grown by hot wire chemical vapor deposition.•Nucleation of α-Sn particles is main mechanism of strain relaxation in GeSn layers.•β-Sn islands forming on GeSn layers due to Sn segregation were identified by CRM. We report on the investigation of the growth defect formation in the GeSn/Ge/Si(001) epitaxial layers grown by hot wire chemical vapor deposition of Ge with simultaneous co-evaporation of Sn from an effusion cell. The nucleation of α-Sn nanoinclusions in the course of the growth process was found to be the main mechanism of elastic strain relaxation in the Ge1-xSnx epitaxial layers at x>2.5% as opposed to the misfit dislocation formation at the GeSn/Ge interfaces. Also, β-Sn nanoislands nucleating on the surface of the GeSn epitaxial layers with increased Sn concentration due to Sn segregation were identified by confocal Raman microscopy.