Quantum signatures of the optomechanical instability

In the past few years, coupling strengths between light and mechanical motion in optomechanical setups have improved by orders of magnitude. Here we show that, in the standard setup under continuous laser illumination, the steady state of the mechanical oscillator can develop a nonclassical, strongly negative Wigner density if the optomechanical coupling is comparable to or larger than the optical decay rate and the mechanical frequency. Because of its robustness, such a Wigner density can be mapped using optical homodyne tomography. This feature is observed near the onset of the instability towards self-induced oscillations. We show that there are also distinct signatures in the photon-photon correlation function g(2)(t) in that regime, including oscillations decaying on a time scale not only much longer than the optical cavity decay time but even longer than the mechanical decay time.