Constrained matched filtering for extended dynamic range and improved noise rejection for Shack-Hartmann wavefront sensing

We recently introduced matched filtering in the context of astronomical Shack-Hartmann wavefront sensing with elongated sodium laser beacons [Appl. Opt. 45, 6568 (2006)]. Detailed wave optics Monte Carlo simulations implementing this technique for the Thirty Meter Telescope dual conjugate adaptive optics system have, however, revealed frequent bursts of degraded closed loop residual wavefront error [Proc. SPIE 6272, 627236 (2006)]. The origin of this problem is shown to be related to laser guide star jitter on the sky that kicks the filter out of its linear dynamic range, which leads to bursts of nonlinearities that are reconstructed into higher-order wavefront aberrations, particularly coma and trifoil for radially elongated subaperture spots. An elegant reformulation of the algorithm is proposed to extend its dynamic range using a set of linear constraints while preserving its improved noise rejection and Monte Carlo performance results are reported that confirm the benefits of the method.