Cooperative “folding transition” in the sequence space facilitates function-driven evolution of protein families

•Natural proteins are classified into well-defined domain families with unique native folds whereas most random polypeptides are unable to fold, implying a “folding transition” in the protein sequence space.•By simulations of a statistical mechanical model of protein sequence alignment, a cooperative two-state “folding transition” in the sequence space is demonstrated for 15 diverse protein domain families.•Well-conserved residues including functionally important sites are found to cooperatively enhance the pattern formation of natural-like sequences.•The cooperative effect of functionally important residues may facilitate the emergence of protein families by natural selection.•The lack of cooperativity in conventional sequence models implies shortcomings of widely used multiple sequence alignments. In the protein sequence space, natural proteins form clusters of families which are characterized by their unique native folds whereas the great majority of random polypeptides are neither clustered nor foldable to unique structures. Since a given polypeptide can be either foldable or unfoldable, a kind of “folding transition” is expected at the boundary of a protein family in the sequence space. By Monte Carlo simulations of a statistical mechanical model of protein sequence alignment that coherently incorporates both short-range and long-range interactions as well as variable-length insertions to reproduce the statistics of the multiple sequence alignment of a given protein family, we demonstrate the existence of such transition between natural-like sequences and random sequences in the sequence subspaces for 15 domain families of various folds. The transition was found to be highly cooperative and two-state-like. Furthermore, enforcing or suppressing consensus residues on a few of the well-conserved sites enhanced or diminished, respectively, the natural-like pattern formation over the entire sequence. In most families, the key sites included ligand binding sites. These results suggest some selective pressure on the key residues, such as ligand binding activity, may cooperatively facilitate the emergence of a protein family during evolution. From a more practical aspect, the present results highlight an essential role of long-range effects in precisely defining protein families, which are absent in conventional sequence models.