For a parameterized hyperbolic system $u_{i+1} = f(u_i,s)$, the derivative of an ergodic average $\langle J\rangle = \lim_{n\rightarrow\infty} \frac1n \sum_1^n J(u_i,s)$ to the parameter $s$ can be computed via the least squares shadowing method. This method solves a constrained least squares problem and computes an approximation to the desired derivative $\frac{d\langle J\rangle}{ds}$ from the solution. This paper proves that as the size of the least squares problem approaches infinity, the computed approximation converges to the true derivative.United States. Air Force Office of Scientific Research (STTR contract FA9550-12-C-0065)United States. National Aeronautics and Space Administration