Newton-Raphson Method

What the Newton–Raphson method does

Closely related Newton updates also locate stationary points of likelihoods and objective functions. This page explains the tangent-line derivation, convergence, worked calculation, multivariable form, optimisation uses and safeguards needed in statistical and health-economic applications.

The method can converge extremely quickly near a suitable solution, but it is not guaranteed to converge from every starting value. Its reliability depends on the function, derivatives, scaling, initial value and any constraints on the parameter space.

The update follows the tangent line to a root

Suppose the aim is to solve:

$$f(x)=0$$

At current iterate (x_k), a first-order Taylor approximation is:

$$f(x) \approx f(x_k)+f'(x_k)(x-x_k)$$

Setting the approximation equal to zero and solving for (x) gives the Newton–Raphson update:

$$x_{k+1} =x_k- \frac{f(x_k)}{f'(x_k)}$$

Geometrically, the algorithm finds where the tangent line at ((x_k,f(x_k))) crosses the horizontal axis. It then repeats the construction from that new point.

A worked example finds the square root of two

To calculate (\sqrt{2}), solve:

$$f(x)=x^2-2=0$$

with derivative:

$$f'(x)=2x$$

The update becomes:

$$x_{k+1} =x_k- \frac{x_k^2-2}{2x_k} =\frac{1}{2} \left(x_k+\frac{2}{x_k}\right)$$

Starting from (x_0=1.5):

The iterates rapidly approach:

$$\sqrt{2}\approx1.4142135624$$

The table demonstrates fast local convergence, not a guarantee that every equation or starting value will behave similarly.

Convergence can be quadratic near a simple root

Let (r) be a simple root, so (f(r)=0) and (f'(r)\ne0). If the function is sufficiently smooth and an iterate is close enough to (r), the error (e_k=x_k-r) approximately satisfies:

$$e_{k+1} \approx \frac{f''(r)}{2f'(r)}e_k^2$$

This quadratic convergence means the number of correct digits can roughly double at each iteration once the algorithm is in its local convergence region. Before that region is reached, steps may be slow, erratic or divergent.

For a root of multiplicity (m>1), ordinary Newton–Raphson generally converges only linearly. If the multiplicity is known, a modified update is:

$$x_{k+1} =x_k-m\frac{f(x_k)}{f'(x_k)}$$

The modification should not be used with an assumed multiplicity that has not been established.

The starting value can determine the destination

Newton–Raphson is a local method. A function with several roots may lead different starting values to different solutions, and some starting points can generate cycles or escape to large values. A derivative near zero can create an extremely large step even when the function value is modest.

Before iterating, an analyst should examine:

• The feasible domain and any parameter boundaries.
• The number and approximate location of possible roots.
• Discontinuities, non-differentiable points and flat regions.
• Whether a bracket with opposite endpoint signs can be found.
• Whether the chosen starting value is supported by theory, data or a coarser search.

A plot or grid evaluation is often a valuable diagnostic in one dimension, but it does not replace numerical checks at the final solution.

Stopping requires more than a small step

An algorithm needs a convergence rule and a maximum iteration count. A small change in (x) can occur because the solution has been reached, because the derivative and scaling produce a tiny step, or because floating-point rounding has stalled the iteration. Multiple criteria give stronger evidence.

Common checks include:

$$|f(x_k)|\le\tau_f$$

and:

|x_{k+1}-x_k| \le \tau_x\left(1+|x_{k+1}| ight)

The tolerances (\tau_f) and (\tau_x) should reflect scale and decision needs. The algorithm should also detect non-finite values, an unusably small derivative, cycling and failure to converge within the permitted iterations.

Bracketing methods trade speed for reliability

Bisection begins with an interval ([a,b]) satisfying (f(a)f(b)<0) for a continuous function and repeatedly halves the interval. It converges more slowly than a successful Newton iteration but maintains a root bracket. The secant method replaces the derivative with a slope calculated from two iterates.

Hybrid solvers use a Newton or interpolation step when it is safe and fall back to a bracket-preserving step otherwise. They are often preferable for production root finding.

Damping can prevent an excessively aggressive step

The full Newton step may overshoot, enter an invalid region or worsen the objective. A damped update multiplies the step by (\alpha_k), where (0<\alpha_k\le1):

$$x_{k+1} =x_k- \alpha_k \frac{f(x_k)}{f'(x_k)}$$

A line search can choose (\alpha_k) to obtain sufficient improvement while respecting the domain. Trust-region methods instead restrict the step to a region where the local approximation is considered reliable. These safeguards trade some ideal local speed for better global behaviour.

Multivariable Newton solves a linear system at each iteration

For a vector function (F:\mathbb{R}^p\rightarrow\mathbb{R}^p), the target is:

$$F(\theta)=0$$

Let (J_F(\theta_k)) be the Jacobian matrix at the current iterate. The Newton step (s_k) solves:

$$J_F(\theta_k)s_k=-F(\theta_k)$$

and updates:

$$\theta_{k+1}=\theta_k+s_k$$

Software should solve the linear system for (s_k) rather than explicitly calculate (J_F(\theta_k)^{-1}). A singular or ill-conditioned Jacobian can make the step non-unique, unstable or uninformative.

Newton optimisation uses the gradient and Hessian

To minimise a twice-differentiable objective (Q(\theta)), Newton's method solves the stationarity equation (\nabla Q(\theta)=0). The Hessian (H_Q(\theta)) is the Jacobian of the gradient, giving the update:

$$H_Q(\theta_k)s_k=-\nabla Q(\theta_k)$$ $$\theta_{k+1}=\theta_k+s_k$$

Equivalently, when the Hessian is invertible:

$$\theta_{k+1} =\theta_k- H_Q(\theta_k)^{-1}\nabla Q(\theta_k)$$

The inverse form explains the mathematics, but a linear solve is preferable computationally. For a local minimum, the Hessian should be positive definite near the solution; otherwise the Newton direction may lead toward a maximum or saddle point.

Maximum-likelihood estimation applies Newton to the score

For log-likelihood (\ell(\theta)), the score vector is:

$$U(\theta)=\nabla\ell(\theta)$$

and the observed Hessian is:

$$H(\theta)=\nabla^2\ell(\theta)$$

Solving (U(\theta)=0) gives the Newton–Raphson likelihood update:

$$\theta_{k+1} =\theta_k- H(\theta_k)^{-1}U(\theta_k)$$

Fisher scoring replaces the negative observed Hessian with the expected information matrix. Iteratively reweighted least squares used for many generalised linear models is closely related. Convergence to a stationary point does not prove that it is the global maximum or that the statistical model is identifiable.

Derivatives can be analytic, automatic or numerical

Accurate derivatives are central to Newton–Raphson. Analytic derivatives can be efficient and precise but are vulnerable to algebra or coding mistakes. Automatic differentiation evaluates derivatives from program operations, while finite differences approximate them through nearby function evaluations.

A centred finite-difference derivative is:

$$f'(x) \approx \frac{f(x+h)-f(x-h)}{2h}$$

Choosing (h) is a balance: a large step creates truncation error, while a very small step amplifies cancellation and floating-point error. Numerical Hessians can be especially noisy. Derivative checks should compare independent methods at several points rather than only at the final solution.

Constraints require transformations or constrained methods

An unconstrained Newton step can generate impossible parameter values, such as negative rates, probabilities outside ([0,1]) or invalid covariance matrices. Simply clipping the proposed value at a boundary changes the algorithm and can cause false convergence.

Possible approaches include:

• Transforming a positive parameter using (\theta=e^\eta).
• Transforming a probability using a logit parameterisation.
• Using a softmax transformation for probabilities that must sum to one.
• Applying a Cholesky parameterisation for positive-definite covariance matrices.
• Using a constrained optimiser with explicit bounds or nonlinear constraints.

Transformations change derivatives and may change numerical conditioning. Results should be translated back to the natural parameter scale with uncertainty handled consistently.

Scaling affects numerical performance

When parameters or equations operate on very different magnitudes, one component can dominate the step or stopping rule. Poor scaling can make the Jacobian or Hessian ill-conditioned and can produce misleading evidence of convergence.

Useful measures include:

• Rescaling parameters to comparable numerical magnitudes.
• Scaling equations so residual components have comparable influence.
• Using dimensionless or standardised parameterisations when scientifically appropriate.
• Applying preconditioning or a parameter-specific step metric.
• Checking condition numbers and linear-solve residuals.

Scaling should improve numerical behaviour without changing the substantive model or hiding weak identification.

Health-economic models use Newton updates in several places

The method often operates inside statistical or modelling software rather than appearing as a visible health-economic output. Understanding it helps analysts diagnose non-convergence and distinguish numerical failure from evidence or structural problems.

Applications include:

• Estimating generalised linear models for cost and utilisation data.
• Fitting parametric survival and competing-risk models.
• Calibrating model parameters to epidemiological or clinical targets.
• Solving break-even or threshold equations such as (\operatorname{INMB}(x)=0).
• Estimating mixed, latent-variable and evidence-synthesis models.
• Finding internal rates, equilibrium conditions or constrained resource-allocation solutions.
• Computing modes in Bayesian approximations and optimisation-based inference.

For a threshold problem, every reported root should be checked against the feasible parameter range and substituted back into the original economic equation.

A threshold example shows the decision use

Suppose incremental net monetary benefit depends nonlinearly on a price (p):

$$\operatorname{INMB}(p) =8000-20p-0.01p^2$$

The break-even price solves (\operatorname{INMB}(p)=0), with derivative:

$$\operatorname{INMB}'(p)=-20-0.02p$$

Starting from (p_0=300), the first update is:

$$p_1 =300- \frac{8000-20(300)-0.01(300)^2} {-20-0.02(300)} =342.3077$$

Further iteration approaches the positive root of approximately £341.64. The quadratic also has a negative mathematical root, but a negative price may be outside the feasible decision problem and should not be reported as an actionable threshold.

Spreadsheet implementation should expose every iteration

A spreadsheet can demonstrate Newton–Raphson by placing the iteration number, current value, function, derivative, proposed step and next value in separate columns. The first row contains the starting value, and each later row references the prior iterate. This visible structure helps identify divergence, oscillation and a derivative approaching zero.

Spreadsheet safeguards should include:

• A fixed maximum iteration count.
• Separate absolute and relative convergence checks.
• A minimum acceptable derivative magnitude.
• Domain checks before every function evaluation.
• A warning for non-finite values or increasing residuals.
• Retention of full calculation precision until presentation.
• Independent substitution of the final value into the original equation.

Circular references and automatic iterative calculation should not be used as an undocumented substitute for an explicit algorithm.

Common failures require diagnosis, not forced convergence

Newton–Raphson can return a number even when the number is not the intended solution. A defensible implementation records the path, final residual and reason for stopping.

• A zero or near-zero derivative can create an undefined or enormous step.
• A poor starting value can lead to divergence, cycling or the wrong root.
• A full step can cross a domain boundary or worsen an objective.
• An inaccurate numerical derivative can point in the wrong direction.
• An indefinite Hessian can lead towards a saddle point or maximum during minimisation.
• A singular Jacobian can indicate redundant equations or non-identifiable parameters.
• A small step can reflect floating-point stagnation rather than a small residual.
• A small gradient can occur on a flat surface far from a useful optimum.
• Convergence from only one start can conceal alternative roots or local optima.
• Tightening tolerances cannot fix a misspecified or ill-conditioned problem.

A defensible Newton–Raphson workflow

A reliable analysis combines the fast local update with domain knowledge, diagnostics and a safer fallback. The workflow should make it possible to distinguish successful mathematical convergence from a merely terminated sequence of numbers.

1. Define the target: State the original root, score or optimisation equation and the feasible parameter domain.
2. Derive the derivatives: Obtain and independently check the derivative, gradient, Jacobian or Hessian.
3. Choose starting values: Use theory, a grid, prior estimates or a bracket, and test multiple starts when several solutions are possible.
4. Scale the problem: Rescale parameters and equations to reduce avoidable conditioning problems.
5. Set safeguards: Specify damping, line search, trust region, derivative thresholds, bounds and a fallback method.
6. Set stopping rules: Require appropriate residual, step and iteration criteria rather than one tolerance alone.
7. Monitor iterations: Record function values, step sizes, domain violations and conditioning diagnostics.
8. Verify the solution: Substitute the estimate into the original equation and check first- and second-order conditions where relevant.
9. Test robustness: Repeat with alternative starts, tolerances, derivative methods and a slower reliable solver.
10. Report transparently: Document the algorithm, initial values, transformations, convergence status, final residual and unresolved numerical limitations.

Newton–Raphson is powerful because it converts local curvature into rapid progress. Its speed becomes trustworthy only when the final solution is feasible, independently verified and robust to reasonable numerical choices.