Mitigating parametric instability in optical gravitational wave detectors

Achieving quantum limited sensitivity in a laser interferometric gravitational wave detector can be hindered by an optomechanical parametric instability of the interferometer. This instability is sustained by a large number of idle high-finesse Stokes modes supported by the system. We show that by optimizing the geometrical shape of the mirrors of the detector, one reduces the diffraction-limited finesse of unessential optical modes and effectively increases the instability threshold. Utilizing parameters of the Advanced LIGO system as a reference, we find that the proposed technique allows constructing a Fabry-Perot interferometer with round-trip diffraction loss of the fundamental mode not exceeding 5 ppm, whereas the loss of the first dipole as well as the other high-order modes exceeds 1000 ppm and 8000 ppm, respectively. This is 2 orders of magnitude higher if compared with a conventional Advanced LIGO interferometer. The optimization comes at the price of tighter tolerances on the mirror tilt stability, but it does not result in a significant modification of the optical beam profile and does not require changes in the gravity detector readout system. The cavity with proposed mirrors is also stable with respect to the slight modification of the mirror shape.