Three dimensional memory device with epitaxial semiconductor pedestal for peripheral transistors

An alternating stack of insulator layers and spacer material layers is formed over a substrate. Stepped surfaces are formed in a contact region in which contact via structures are to be subsequently formed. An epitaxial semiconductor pedestal can be formed by a single epitaxial deposition process that is performed after formation of the stepped surfaces and prior to formation of memory openings, or a combination of a first epitaxial deposition process performed prior to formation of memory openings and a second epitaxial deposition process performed after formation of the memory openings. The epitaxial semiconductor pedestal can have a top surface that is located above a topmost surface of the alternating stack. The spacer material layers are formed as, or can be replaced with, electrically conductive layers. Backside contact via structures can be subsequently formed.