Statistical Methods
Check an Analytic Gradient Before Trusting an Optimizer
Define the task
An optimizer can converge to a misleading result if the supplied gradient does not differentiate the objective actually evaluated. Check signs, parameter ordering, transformations and penalty terms before relying on a convergence message. A finite-difference gradient comparison is a diagnostic, not proof that the objective is scientifically appropriate or globally optimized. Use synthetic inputs and parameter values with known behavior before applying the implementation to an analysis.
Carry out the check
Compare the analytic gradient with forward finite differences at several interior points and over a range of step sizes. Keep function evaluations and gradients on the same scale. Too small a step can amplify roundoff; too large a step increases approximation error. Near constraints or nondifferentiable points, an ordinary forward perturbation may leave the valid domain or fail to represent a derivative. Record absolute discrepancies and their scale rather than accepting one unexplained scalar threshold for every objective.
Worked hypothetical example
For a hypothetical f(x)=x^2 at x=3, the analytic derivative is 6. A forward difference with h=0.001 gives [(3.001)^2−9]/0.001=6.001 in exact arithmetic, a discrepancy of 0.001 from truncation alone. An incorrectly supplied derivative of 3 would differ by about 3.001. This example separates an implementation error from expected finite-step error. Test multiple coordinates and parameter values, document unresolved discrepancies, and rerun the check whenever objective or transformation code changes.
Method checked 19 November 2026 against Primary method documentation: check_grad.