Jacobian & Hessian Matrices for Deep Learning

Introduction

In single-variable calculus, we have f'(x) (first derivative/slope) and f''(x) (second derivative/curvature). But neural networks operate on vectors with millions of dimensions.

When we generalize derivatives to vectors:

Jacobian (First Order)

Matrix of all first partial derivatives. Generalizes gradient to vector-valued functions.

Hessian (Second Order)

Matrix of all second partial derivatives. Captures curvature information.

The Derivative Hierarchy

The type of derivative depends on the input and output dimensions:

The Jacobian Matrix

For a function \left. \mathbf{f}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{m} \right. (n inputs, m outputs), the Jacobian is an m x n matrix:

J = \begin{bmatrix} \frac{\partial f_{1}}{\partial x_{1}} & \frac{\partial f_{1}}{\partial x_{2}} & \cdots & \frac{\partial f_{1}}{\partial x_{n}} \\ \frac{\partial f_{2}}{\partial x_{1}} & \frac{\partial f_{2}}{\partial x_{2}} & \cdots & \frac{\partial f_{2}}{\partial x_{n}} \\ {\vdots} & {\vdots} & \ddots & {\vdots} \\ \frac{\partial f_{m}}{\partial x_{1}} & \frac{\partial f_{m}}{\partial x_{2}} & \cdots & \frac{\partial f_{m}}{\partial x_{n}} \end{bmatrix}J=​∂x1​∂f1​​∂x1​∂f2​​⋮∂x1​∂fm​​​∂x2​∂f1​​∂x2​∂f2​​⋮∂x2​∂fm​​​⋯⋯⋱⋯​∂xn​∂f1​​∂xn​∂f2​​⋮∂xn​∂fm​​​​

Row i = how output i changes with each input. Column j = how each output changes with input j.

Geometric Meaning

The Jacobian represents the best linear approximation to f near a point. It tells us how a small change in input \delta x affects the output:

\mathbf{f}(\mathbf{x} + \delta\mathbf{x}) \approx \mathbf{f}(\mathbf{x}) + J \cdot \delta\mathbf{x}

Jacobian: Worked Example

Consider a function from \left. \mathbb{R}^{2}\rightarrow\mathbb{R}^{2} \right.

\mathbf{f}(x,y) = \begin{bmatrix} {x^{2} + y} \\ {xy} \end{bmatrix}f(x,y)=[x2+yxy​]

Step 1: Compute all partial derivatives

\frac{\partial f_{1}}{\partial x} = 2x

\frac{\partial f_{1}}{\partial y} = 1

\frac{\partial f_{2}}{\partial x} = y

\frac{\partial f_{2}}{\partial y} = x

Step 2: Assemble the Jacobian

J = \begin{bmatrix} {2x} & 1 \\ y & x \end{bmatrix}J=[2xy​1x​]

Step 3: Evaluate at a point (x=2, y=3)

J\mid_{(2,3)} = \begin{bmatrix} 4 & 1 \\ 3 & 2 \end{bmatrix}J∣(2,3)​=[43​12​]

Interactive: Jacobian in Action

See how the Jacobian provides a linear approximation to how outputs change with inputs. Move the base point and perturbation to explore.

Jacobian Linearization

\mathbf{f}(x,y) = (x^{2} + y,xy) Drag the points to explore.

Local Grid

Reset

Input Space (x, y)

Base: (1.50, 1.00)

Perturbation \delta (0.50, 0.30)

Output Space (u, v)

Approximation Error0.2915

Jacobian J at Base

\begin{bmatrix} 3.00 & 1.00 \\ 1.00 & 1.50 \end{bmatrix}[3.001.00​1.001.50​]

True Change

Linear Approx

The Hessian Matrix

For a scalar-valued function \left. f:\mathbb{R}^{n}\rightarrow\mathbb{R} \right. (like a loss function), the Hessian is the n x n matrix of second-order partial derivatives:

H = \nabla^{2}f = \begin{bmatrix} \frac{\partial^{2}f}{\partial x_{1}^{2}} & \frac{\partial^{2}f}{\partial x_{1}\partial x_{2}} & \cdots \\ \frac{\partial^{2}f}{\partial x_{2}\partial x_{1}} & \frac{\partial^{2}f}{\partial x_{2}^{2}} & \cdots \\ {\vdots} & {\vdots} & \ddots \end{bmatrix}H=∇2f=​∂x12​∂2f​∂x2​∂x1​∂2f​⋮​∂x1​∂x2​∂2f​∂x22​∂2f​⋮​⋯⋯⋱​​

Symmetric

For continuous second partials:

\frac{\partial^{2}f}{\partial x_{i}\partial x_{j}} = \frac{\partial^{2}f}{\partial x_{j}\partial x_{i}}

So H = H^{T}

Curvature

The Hessian captures how the gradient itself changes. It describes the "bowl shape" of the function.

Hessian: Worked Example

Consider a loss function f(x,y) = x^{2} + 3xy + y^{2}

Step 1: First partial derivatives (gradient)

\nabla f = \begin{bmatrix} {2x + 3y} \\ {3x + 2y} \end{bmatrix}∇f=[2x+3y3x+2y​]

Step 2: Second partial derivatives

\frac{\partial^{2}f}{\partial x^{2}} = 2

\frac{\partial^{2}f}{\partial x\partial y} = 3

\frac{\partial^{2}f}{\partial y\partial x} = 3

\frac{\partial^{2}f}{\partial y^{2}} = 2

Step 3: Assemble Hessian

H = \begin{bmatrix} 2 & 3 \\ 3 & 2 \end{bmatrix}H=[23​32​]

Note: constant because f is quadratic.

Interactive: Curvature & Critical Points

Adjust the Hessian eigenvalues to see how they determine the shape of the loss surface and classify critical points.

Hessian Curvature & Step Size

High curvature = Steep walls = Inverse Hessian prevents overshooting.

Gradient Descent

Newton's Method

Flat (Low Curvature)Steep (High Curvature)

Hessian f^{\prime\prime}(x) = 2.0

Step Size Analysis

Newton Step\Delta x = - f^{\prime}/f^{\prime\prime}

Adapts to curvature. If curve is steep (high f^{\prime\prime} step is scaled down.

Size: 2.00Perfect

Gradient Descent\Delta x = - \eta f^{\prime}

Fixed learning rate. Ignores curvature.

Size: 0.80Too Slow

Key Insight

The Hessian H = \nabla^{2}f acts as a "smart scaling matrix".

In steep directions (high curvature), H^{- 1} shrinks the gradient to prevent overshooting. In flat directions, it expands the step to speed up.

Critical Points: The Eigenvalue Test

At a critical point (where gradient = 0), the Hessian's eigenvalues tell us the nature of that point:

All positive

Function curves UP in all directions. Local minimum.

All negative

Function curves DOWN in all directions. Local maximum.

Mixed signs

Curves up in some directions, down in others. Saddle point.

Why This Matters

In high-dimensional neural network loss landscapes, saddle points are far more common than local minima. Understanding the Hessian helps explain why optimization can stall and why momentum-based methods help.

ML Applications

Backpropagation = Jacobian-Vector Products

When computing gradients through a neural network, each layer contributes a Jacobian. The chain rule becomes: \nabla_{x}L = J_{1}^{T}J_{2}^{T}\cdots J_{n}^{T}\nabla_{y}L

Newton's Method

Second-order optimization: \theta_{new} = \theta - H^{- 1}\nabla f Uses curvature to take smarter steps. O(n³) to compute.

Hessian-Free Optimization

Clever algorithms that use Hessian information without computing the full matrix. Conjugate gradient methods can compute Hessian-vector products efficiently.

Loss Landscape Analysis

Researchers study Hessian eigenvalue distributions. Sharp minima (large eigenvalues) tend to generalize worse than flat minima (small eigenvalues).