Vanishing Gradients

Finding Structure in Time (Vanilla RNN)

Easy

Problem

Theory

The vanishing gradient problem is the central failure mode that limits Vanilla RNNs from learning long-range dependencies. When gradients propagate backward through many time steps during Backpropagation Through Time (BPTT), they involve repeated multiplication by the recurrent weight matrix W_{hh}. The spectral norm of this matrix -- its largest singular value -- determines whether gradients decay to zero, explode to infinity, or remain stable.

What It Is

In a Vanilla RNN, the hidden state at time t is computed as h_t = \tanh(W_{hh} h_{t-1} + W_{xh} x_t + b). During training, the loss L is computed at the final time step (or accumulated across steps), and gradients must flow backward through the entire sequence. To update parameters based on information from early time steps, the gradient of the loss with respect to an early hidden state h_0 must pass through every intermediate time step.

This backward pass involves a chain of matrix multiplications. At each step, the gradient is multiplied by the Jacobian of the hidden state transition, which in a simplified model reduces to multiplication by W_{hh}. Over T steps, the gradient accumulates T such multiplications. The behavior of this repeated multiplication is entirely governed by the spectral properties of W_{hh}.

This problem demonstrates that behavior in its purest form: starting from gradient norm 1.0 at the current time step, compute what happens to the gradient norm at each of T backward steps. The result reveals whether the network can propagate useful gradient information across the full sequence or whether the signal is destroyed long before it reaches the earliest time steps.

Key Equations

The gradient of the loss with respect to an early hidden state h_k depends on the gradient at the final step h_T through the chain rule:

\frac{\partial L}{\partial h_k} = \frac{\partial L}{\partial h_T} \prod_{t=k+1}^{T} \frac{\partial h_t}{\partial h_{t-1}}

In the simplified model where the tanh derivative is approximately 1, each Jacobian \frac{\partial h_t}{\partial h_{t-1}} reduces to W_{hh}. The product becomes:

\prod_{t=k+1}^{T} \frac{\partial h_t}{\partial h_{t-1}} \approx W_{hh}^{T-k}

Taking the norm of both sides:

\left\|\frac{\partial L}{\partial h_k}\right\| \propto \|W_{hh}\|_2^{\,T-k}

where \|W_{hh}\|_2 is the spectral norm -- the largest singular value. Starting from norm 1.0 and propagating backward through T steps, the gradient norm at step i is:

g_i = \|W_{hh}\|_2^{\,i}

The Spectral Norm

The spectral norm of a matrix A is defined as its largest singular value, denoted \sigma_1(A) or \|A\|_2. It has a precise geometric meaning: the maximum factor by which A can stretch any unit vector.

\|A\|_2 = \max_{\|v\| = 1} \|Av\| = \sigma_1(A)

To compute the spectral norm, perform a Singular Value Decomposition (SVD) of W_{hh} = U \Sigma V^T, where \Sigma contains singular values \sigma_1 \geq \sigma_2 \geq \ldots \geq \sigma_n \geq 0. The spectral norm is \sigma_1, the largest entry.

The spectral norm governs the worst-case behavior of repeated multiplication. When the gradient vector is multiplied by W_{hh} repeatedly, the component aligned with the top singular vector gets scaled by \sigma_1 at each step. After T multiplications, the gradient norm scales as \sigma_1^T.

The spectral norm is distinct from the Frobenius norm \|A\|_F = \sqrt{\sum_{i,j} A_{ij}^2}, which measures total matrix magnitude but does not predict gradient behavior under repeated multiplication. A matrix can have a large Frobenius norm but spectral norm less than 1, leading to vanishing gradients despite the matrix "looking large."

Three Regimes

The value of \|W_{hh}\|_2 relative to 1 determines which of three qualitatively different regimes the gradient falls into.

Vanishing (\|W_{hh}\|_2 < 1). Each backward step shrinks the gradient. With \sigma_1 = 0.8 and T = 10: the gradient norms are [0.8, 0.64, 0.512, 0.4096, 0.3277, 0.2621, 0.2097, 0.1678, 0.1342, 0.1074]. By step 10, the gradient retains roughly $$ of its original magnitude. By step 50, it would be $ \approx 1.4 \times 10^{-5}$ -- effectively zero.

Stable (\|W_{hh}\|_2 = 1). The gradient norm remains constant: 1^T = 1 for all T. Every time step contributes equally to learning regardless of sequence length. Achieving this requires explicit constraints such as orthogonal initialization or spectral normalization.

Exploding (\|W_{hh}\|_2 > 1). Each backward step amplifies the gradient. With \sigma_1 = 1.2 and T = 10: the gradient norms are [1.2, 1.44, 1.728, 2.0736, 2.4883, 2.9860, 3.5832, 4.2998, 5.1598, 6.1917]. The gradient grows over 6\times in 10 steps. For $ = 100$, it would reach 1.2^{100} \approx 8.3 \times 10^7.

Why This Matters

Long-range dependencies become invisible. Consider a language model processing "The cat, which sat on the mat near the door by the window, was sleeping." The subject "cat" and verb "was" are separated by many tokens. If gradients vanish over that distance, the model receives no learning signal connecting the verb's error back to the subject's representation. The model learns only short-range patterns.

Training appears to converge but the model is shallow. Vanishing gradients do not cause training to crash. The model still learns local patterns (bigrams, trigrams) because those involve short gradient paths. The loss decreases and metrics improve, but the model never captures long-range structure. This makes vanishing gradients harder to diagnose than exploding gradients, which produce obvious numerical failures.

Exploding gradients destabilize training. Large gradients cause large parameter updates, shifting the loss landscape dramatically. The next forward pass produces very different activations, leading to even larger gradients. This positive feedback loop causes the loss to spike, oscillate, or diverge. NaN values in parameters are a typical symptom.

The Exponential Nature

Even small deviations from spectral norm 1 compound rapidly. This is not gradual degradation -- it is exponential, and exponentials are deceptively fast.

A spectral norm of 0.9 -- only 10\% below the stable threshold -- yields: 0.9^{10} \approx 0.349, 0.9^{50} \approx 0.00515, 0.9^{100} \approx 2.66 \times 10^{-5}. A sequence length of 100 tokens is routine in NLP, yet even this "almost stable" spectral norm destroys gradient information completely.

The same exponential works in reverse. A spectral norm of 1.1 yields: 1.1^{10} \approx 2.594, 1.1^{50} \approx 117.4, 1.1^{100} \approx 13{,}781. The gradient is amplified by nearly 14{,}000 over 100 steps.

This exponential sensitivity explains why Vanilla RNNs fail on any task requiring memory beyond 10-$$ time steps. The spectral norm would need to be maintained at exactly 1.0 throughout training, but gradient descent has no mechanism to enforce this constraint. As weights update, the spectral norm drifts, and the exponential takes over.

Paper Context

The vanishing gradient problem was formally identified in two foundational works. Sepp Hochreiter's 1991 diploma thesis first demonstrated that gradient flow in recurrent networks is fundamentally unstable, showing that error signals flowing backward in time either shrink or grow exponentially.

Bengio, Simard, and Frasconi formalized this in their 1994 paper "Learning Long-Term Dependencies with Gradient Descent is Difficult." They proved that for recurrent networks with bounded weights, the gradient must either vanish or explode exponentially with temporal distance. The paper states: "The influence of a given input on the hidden layer, and therefore on the network output, either decays or blows up exponentially as it cycles around the network's recurrent connections." This is a mathematical inevitability for networks that propagate gradients through repeated matrix multiplication.

Bengio et al. demonstrated that the problem is not specific to any activation function or training algorithm. As long as the network relies on multiplying Jacobians through time, the exponential dynamics are unavoidable. The only escape is to change the architecture -- to create paths through which gradients flow without repeated multiplication by the same matrix.

This analysis directly motivated the Long Short-Term Memory (LSTM) by Hochreiter and Schmidhuber (1997). The LSTM's cell state with additive updates controlled by gates creates a gradient highway through time. The forget gate, when close to 1, allows the gradient to pass through nearly unchanged: \frac{\partial c_t}{\partial c_{t-1}} = f_t, replacing the problematic W_{hh}^T with scalar gate values and breaking the exponential decay.

Numerical Example

Consider T = 10 time steps with three different spectral norms: 0.8, 1.0, and 1.2. Starting from gradient norm 1.0, the gradient norm at step i is \sigma_1^i.

Spectral norm = 0.8 (vanishing):

Step 1: 0.8^1 = 0.8000. Step 2: $ = 0.6400$. Step 3: $ = 0.5120$. Step 4: $ = 0.4096$. Step 5: $ = 0.3277$. Step 6: $ = 0.2621$. Step 7: $ = 0.2097$. Step 8: $ = 0.1678$. Step 9: $ = 0.1342$. Step 10: $ = 0.1074$.

Spectral norm = 1.0 (stable):

All 10 steps produce gradient norm 1.0. Every time step contributes equally to learning.

Spectral norm = 1.2 (exploding):

Step 1: 1.2^1 = 1.2000. Step 2: $ = 1.4400$. Step 3: $ = 1.7280$. Step 4: $ = 2.0736$. Step 5: $ = 2.4883$. Step 6: $ = 2.9860$. Step 7: $ = 3.5832$. Step 8: $ = 4.2998$. Step 9: $ = 5.1598$. Step 10: $ = 6.1917$.

The contrast is stark: after 10 steps, the vanishing case retains 10.7\% of the gradient while the exploding case has amplified it to 619\%. Extend to T = 50: 0.8^{50} \approx 1.4 \times 10^{-5} versus 1.2^{50} \approx 9100. The divergence between regimes grows dramatically with sequence length.

Solutions

LSTM constant error carousel. The LSTM cell state uses additive updates: c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t. The gradient of c_t with respect to c_{t-1} is f_t (element-wise), not a full matrix multiplication. When the forget gate is near 1, the gradient passes through almost unchanged, creating a "constant error carousel" that does not suffer from exponential decay.

GRU gating. The Gated Recurrent Unit (Cho et al., 2014) uses the update gate z_t to interpolate: h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t. When z_t \approx 0, h_t \approx h_{t-1} and the gradient flows through unchanged. The GRU merges forget and input gates into one, reducing parameters while still providing gradient-stable paths.

Gradient clipping. Pascanu, Mikolov, and Bengio (2013) proposed clipping the gradient norm to a threshold \tau: if \|g\| > \tau, replace g with \tau \cdot g / \|g\|. This handles exploding gradients by capping update magnitudes. It does nothing for vanishing gradients -- a gradient already near zero cannot be helped by clipping.

Orthogonal initialization and residual connections. Initializing W_{hh} as an orthogonal matrix ensures spectral norm 1 at the start of training. Residual connections (y = x + f(x)) in stacked RNNs provide skip paths with an identity gradient term, preventing complete vanishing across layers.

Pitfalls

• Confusing spectral norm with Frobenius norm. The Frobenius norm \|A\|_F = \sqrt{\sum_{ij} A_{ij}^2} measures overall matrix magnitude but does not predict gradient behavior under repeated multiplication. A 100 \times 100 matrix with all singular values equal to 0.5 has Frobenius norm 5.0 but spectral norm 0.5, and gradients vanish. The spectral norm is the correct quantity because it captures the multiplicative growth rate.

• Forgetting that tanh derivative further reduces the gradient. The simplified model ignores the activation function. In practice, each backward step multiplies by \text{diag}(\tanh'(z_t)) \cdot W_{hh}, where \tanh'(z) = 1 - \tanh^2(z) \in (0, 1]. The effective spectral norm per step is strictly less than \|W_{hh}\|_2. Vanishing is worse in practice than the formula suggests.

• Assuming gradient clipping solves vanishing gradients. Clipping prevents norms from exceeding a threshold, handling explosions. It does not amplify small gradients. If the spectral norm is 0.9 and gradients have decayed to 10^{-5} after 100 steps, clipping does nothing. Solving vanishing gradients requires architectural changes, not clipping.

• Ignoring that this is a simplified model. The formula g_i = \sigma_1^i assumes the same W_{hh} at every step, no activation derivative, and alignment with the top singular vector. In a real network, the tanh derivative varies by time step and the gradient direction rotates. The simplified model captures qualitative behavior but the exact rate differs in practice.

• Initializing LSTM forget gate bias at zero. When using an LSTM to solve vanishing gradients, initializing the forget gate bias at 0 gives \sigma(0) = 0.5, halving the cell state at every step -- itself a form of vanishing gradient with rate 0.5^T. Best practice (Gers et al., 2000) is to initialize the bias to $$ or higher so \sigma(1) \approx 0.73, biasing toward retention.

• Conflating vanishing gradients with zero gradients. Vanishing gradients are not literally zero -- they are exponentially small. A gradient of 10^{-15} is technically nonzero but in float32 is indistinguishable from zero. The practical threshold depends on precision and the optimizer: in float32 with Adam's epsilon of $$, gradients below roughly 10^{-7} contribute nothing to updates.

Examples

Hints

1. Use np.linalg.norm(W_hh, ord=2) for the spectral norm.
2. np.arange(T) gives the exponents from zero through T minus one.

Requirements

• Compute the matrix spectral norm with NumPy.
• Return powers from zero through T minus one in order.
• Return a one-dimensional float64 NumPy array.

Constraints

• T is a positive integer.
• W_hh is a finite square float64 matrix.
• This is a linear upper-bound diagnostic, not the complete tanh BPTT gradient.

Starter Code

import numpy as np

def compute_gradient_norm_decay(T: int, W_hh: np.ndarray) -> np.ndarray:
    """
    Returns T float64 spectral-norm powers.
    """
    pass

Test Cases