Semiconductor device and a method of manufacturing the same

In a semiconductor device with a shared contact, a gate electrode is formed via a gate insulating film on a semiconductor substrate and a sidewall insulating film is formed on both side faces of the gate electrode. At least one of the surface parts of the semiconductor substrate adjacent to both sides of the gate electrode is removed beyond the lower part of the sidewall insulating film and to the underside of the gate electrode. Then, the gate insulating film exposed in the remove part is removed. An impurity-doped semiconductor layer is formed in the part where the semiconductor substrate and the gate insulating film have been removed.