Coevolution of cannibalistic predators and timid prey: evolutionary cycling and branching

•We derive a predator-prey model with ecological bistability from individual-level processes.•We apply critical function analysis of adaptive dynamics to treat general classes of trade-off functions between cannibalism and prey capture.•We investigate the coevolution of cannibalistic predators and timid prey.•We find that cannibalism can emerge as an evolutionary response to timidity of the prey.•The long-term coevolutionary outcomes include several kinds of evolutionary cycles, and further diversification of cannibalism through evolutionary branching. We investigate the coevolution of cannibalistic predators and timid prey, which seek refuge upon detecting a predator. To understand how the species affect each other’s evolution, we derived the ecological model from individual-level processes using ordinary differential equations. The ecological dynamics exhibit bistability between equilibrium and periodic attractors, which may disappear through catastrophic bifurcations. Using the critical function analysis of adaptive dynamics, we classify general trade-offs between cannibalism and prey capture that produce different evolutionary outcomes. The evolutionary analysis reveals several ways in which cannibalism emerges as a response to timidity of the prey. The long-term coevolution either attains a singularity, or becomes cyclic through two mechanisms: genetical cycles through Hopf bifurcation of the singularity, or ecogenetical cycles involving abrupt switching between ecological attractors. Further diversification of cannibalism occurs through evolutionary branching, which is predicted to be delayed when simultaneous prey evolution is necessary for the singularity’s attainability. We conclude that predator-prey coevolution produces a variety of outcomes, in which evolutionary cycles are commonplace.