Dynamics near the invariant manifolds after a Hamiltonian-Hopf bifurcation

We consider a one parameter family of 2-DOF Hamiltonian systems having an equilibrium point that undergoes a Hamiltonian-Hopf bifurcation. We briefly review the well-established normal form theory in this case. Then we focus on the invariant manifolds when there are homoclinic orbits to the complex-saddle equilibrium point, and we study the behavior of the splitting of the 2D invariant manifolds. The symmetries of the normal form are used to reduce the dynamics around the invariant manifolds to the dynamics of a family of area-preserving near-identity Poincaré maps that can be extended analytically to a suitable neighborhood of the separatrices. This allows, in particular, to use well-known results for area-preserving maps and derive an explicit upper bound of the splitting of separatrices for the Poincaré map. We illustrate the results in a concrete example. Different Poincaré sections are used to visualize the dynamics near the 2D invariant manifolds. Last section deals with the derivation of a separatrix map to study the chaotic dynamics near the 2D invariant manifolds. •Sokolskii normal form of a Hamiltonian system near a complex saddle fixed point.•Computation of the two dimensional invariant manifolds of complex saddle points.•Derivation and accurate numerical check of the splitting of separatrices formula.•Derivation of a separatrix map to model the dynamics close to the manifolds.