Statistical Methods

Check an Analytic Gradient Before Trusting an Optimizer

Define the task

An optimizer can converge to a misleading result if the supplied gradient does not differentiate the objective actually evaluated. Check signs, parameter ordering, transformations and penalty terms before relying on a convergence message. A finite-difference gradient comparison is a diagnostic, not proof that the objective is scientifically appropriate or globally optimized. Use synthetic inputs and parameter values with known behavior before applying the implementation to an analysis.

Carry out the check

Compare the analytic gradient with forward finite differences at several interior points and over a range of step sizes. Keep function evaluations and gradients on the same scale. Too small a step can amplify roundoff; too large a step increases approximation error. Near constraints or nondifferentiable points, an ordinary forward perturbation may leave the valid domain or fail to represent a derivative. Record absolute discrepancies and their scale rather than accepting one unexplained scalar threshold for every objective.

Worked hypothetical example

For a hypothetical f(x)=x^2 at x=3, the analytic derivative is 6. A forward difference with h=0.001 gives [(3.001)^2−9]/0.001=6.001 in exact arithmetic, a discrepancy of 0.001 from truncation alone. An incorrectly supplied derivative of 3 would differ by about 3.001. This example separates an implementation error from expected finite-step error. Test multiple coordinates and parameter values, document unresolved discrepancies, and rerun the check whenever objective or transformation code changes.

Method checked 19 November 2026 against Primary method documentation: check_grad.

Related reading

Reading path: Analyze data. Guide 182 of 182. Choose any guide to suit your project.

Statistical Methods

Assess sensitivity before trusting a two-variable linear solve

Assess sensitivity of a small linear system to its inputs. Includes a worked matrix example and reproducibility checks.

Statistical Methods

Review Convergence Before Trusting a Fitted Model

Separate a numerical stopping message from evidence that the intended optimum and uncertainty estimates are reliable.

Browse this reading pathTools for this stageGuides and downloads for this stage