Hidden State Update

Learning Phrase Representations using RNN Encoder-Decoder (GRU)

Easy

Problem

Theory

The hidden state update is the final step in a GRU cell, where the new hidden state h_t is formed by linearly interpolating between the previous hidden state h_{t-1} and the candidate activation \tilde{h}_t. Described in Cho et al. (2014), this formula is what allows a GRU to selectively remember or overwrite information at every time step.

What It Is

After the reset gate has produced a candidate activation \tilde{h}_t and the update gate has produced a blending coefficient z_t, the GRU must combine the old hidden state with the new candidate into a single output. The hidden state update performs this combination through linear interpolation: a weighted average where the weights at each dimension sum to exactly one.

The result h_t serves a dual role. It is the output of the GRU cell at time step t (unlike the LSTM, which has separate cell state and hidden state), and it is also the memory carried forward to time step t+1. The interpolation formula simultaneously decides what to output and what to remember.

No additional parameters are needed for the interpolation itself. The update gate z_t was already computed using learned weight matrices. The hidden state update is purely a pointwise arithmetic operation: multiply, subtract, multiply, add. Yet this simple formula is how the GRU learns when to maintain long-term memory and when to incorporate new information.

Key Equations

The Interpolation Formula

h_t = z_t \odot h_{t-1} + (1 - z_t) \odot \tilde{h}_t

where:

• h_t \in \mathbb{R}^H is the new hidden state (output of this time step)
• z_t \in (0, 1)^H is the update gate output (computed earlier in the pipeline)
• h_{t-1} \in \mathbb{R}^H is the previous hidden state (memory from the last time step)
• \tilde{h}_t \in (-1, 1)^H is the candidate hidden state (new information via the reset gate)
• \odot denotes element-wise (Hadamard) multiplication, not matrix multiplication

Element-wise Form

For each hidden dimension k (where k = 1, 2, \ldots, H):

h_t^{(k)} = z_t^{(k)} \cdot h_{t-1}^{(k)} + (1 - z_t^{(k)}) \cdot \tilde{h}_t^{(k)}

Each dimension operates independently. Dimension k has its own gate value z_t^{(k)} controlling how much old value to retain versus how much new candidate to accept.

The Complementary Constraint

z_t^{(k)} + (1 - z_t^{(k)}) = 1 \quad \forall \, k

Because z_t comes from a sigmoid with range (0, 1), both z_t and (1 - z_t) are strictly positive and sum to one. The output h_t is a convex combination of h_{t-1} and \tilde{h}_t, always lying on the line segment between them in each dimension.

Why Linear Interpolation

Smooth Blending Between Old and New

Linear interpolation provides a continuous spectrum between fully keeping the old state (z = 1) and fully replacing with the new candidate (z = 0). Any value in between produces a smooth blend. During training, gradients can nudge z_t incrementally, allowing the network to learn fine-grained memory policies rather than hard binary switches.

Independent Control Per Dimension

Each hidden dimension has its own gate value. In a hidden state of size H = 256, there are 256 independent interpolation decisions at every time step. Dimension 12 might keep 95% of its old value (tracking sentence topic), while dimension 47 replaces 80% (adapting to the current word). This per-dimension independence allows a single hidden state vector to maintain multiple timescales of information.

Gradient Flow Properties

The z_t \odot h_{t-1} term creates a direct linear path from h_{t-1} to h_t. When z_t \approx 1, the gradient \partial h_t / \partial h_{t-1} \approx 1 along that dimension. Dimensions that need to preserve information over many time steps learn to keep z_t close to 1, creating a gradient highway analogous to skip connections in residual networks.

Per-Dimension Independence

The hidden state update is not a single global decision but H independent local decisions. Consider a GRU with H = 4:

• z_t = [0.95, \; 0.10, \; 0.50, \; 0.02]

This means:

• Dimension 0 (z = 0.95): Keep 95% old, take 5% new. Memory mode.
• Dimension 1 (z = 0.10): Keep 10% old, take 90% new. Update mode.
• Dimension 2 (z = 0.50): Equal blend. Smooth transition.
• Dimension 3 (z = 0.02): Near-complete replacement, effectively resetting.

All four behaviors happen simultaneously. The network learns which dimensions serve as long-term memory channels and which should respond to new inputs. If the update gate produced a single scalar for the entire hidden state, all dimensions would update at the same rate, severely limiting expressiveness.

The Complete GRU Pipeline

Step 1: Update Gate

z_t = \sigma(W_z x_t + U_z h_{t-1} + b_z)

Decides per dimension how much of the old state to retain. Output range: (0, 1)^H.

Step 2: Reset Gate

r_t = \sigma(W_r x_t + U_r h_{t-1} + b_r)

Decides how much of the previous hidden state should influence the candidate. When r_t \approx 0, the candidate ignores previous state entirely.

Step 3: Candidate Hidden State

\tilde{h}_t = \tanh(W_h x_t + U_h (r_t \odot h_{t-1}) + b_h)

The "proposed new content." The reset gate modulates how much h_{t-1} participates. Tanh squashes results to (-1, 1).

Step 4: Hidden State Update (This Problem)

h_t = z_t \odot h_{t-1} + (1 - z_t) \odot \tilde{h}_t

Blends old and new. The reset gate controls what goes into the candidate (Step 3), while the update gate controls how much of the candidate reaches the final state (Step 4).

Paper Context

Cho et al. (2014)

In "Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation," the hidden state update appears as Equation 5. The authors state: "The actual activation of the proposed unit h_j^t is then a linear interpolation between the previous activation h_j^{t-1} and the candidate activation \tilde{h}_j^t."

Leaky Integration

The hidden state update is a form of leaky integration from signal processing and neuroscience. In a leaky integrator, the current state is a weighted sum of the previous state (decayed by a leak factor) and new input. The GRU's z_t plays the role of the leak rate, but unlike classical leaky integrators where the rate is fixed, the GRU learns it dynamically as a function of input and state.

Enabling Long-Term Memory

Standard RNNs compute h_t = \tanh(W x_t + U h_{t-1}), completely overwriting the hidden state at each step. The GRU's interpolation formula provides a selective preservation mechanism: by setting z_t close to 1, the network copies information forward across many time steps with minimal degradation, analogous to skip connections in residual networks.

Numerical Example

Setup

GRU with hidden size H = 4, single sample (N = 1):

z_t = [0.8, \; 0.2, \; 0.6, \; 0.1]

h_{t-1} = [1.0, \; -0.5, \; 0.3, \; 2.0]

\tilde{h}_t = [-0.4, \; 0.9, \; 0.7, \; -1.0]

Step 1: Compute z_t \odot h_{t-1}

• Dim 0: 0.8 \times 1.0 = 0.80
• Dim 1: 0.2 \times (-0.5) = -0.10
• Dim 2: 0.6 \times 0.3 = 0.18
• Dim 3: 0.1 \times 2.0 = 0.20

z_t \odot h_{t-1} = [0.80, \; -0.10, \; 0.18, \; 0.20]

Step 2: Compute (1 - z_t)

(1 - z_t) = [0.2, \; 0.8, \; 0.4, \; 0.9]

Step 3: Compute (1 - z_t) \odot \tilde{h}_t

• Dim 0: 0.2 \times (-0.4) = -0.08
• Dim 1: 0.8 \times 0.9 = 0.72
• Dim 2: 0.4 \times 0.7 = 0.28
• Dim 3: 0.9 \times (-1.0) = -0.90

(1 - z_t) \odot \tilde{h}_t = [-0.08, \; 0.72, \; 0.28, \; -0.90]

Step 4: Add Both Terms

• Dim 0: 0.80 + (-0.08) = 0.72
• Dim 1: -0.10 + 0.72 = 0.62
• Dim 2: 0.18 + 0.28 = 0.46
• Dim 3: 0.20 + (-0.90) = -0.70

h_t = [0.72, \; 0.62, \; 0.46, \; -0.70]

Interpreting the Results

• Dim 0 (z = 0.8): Old = 1.0, candidate = -0.4, result = 0.72. High z kept most of the old value.
• Dim 1 (z = 0.2): Old = -0.5, candidate = 0.9, result = 0.62. Low z swung heavily toward the candidate.
• Dim 2 (z = 0.6): Old = 0.3, candidate = 0.7, result = 0.46. Moderate z blended evenly.
• Dim 3 (z = 0.1): Old = 2.0, candidate = -1.0, result = -0.70. Very low z caused near-complete replacement.

Connection to LSTM

LSTM Cell State Update

c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t

The LSTM uses two separate gates: f_t (forget gate) and i_t (input gate), computed by separate weight matrices. f_t + i_t is not constrained to equal 1.

GRU's Complementary Design

h_t = z_t \odot h_{t-1} + (1 - z_t) \odot \tilde{h}_t

The GRU uses a single gate z_t to replace both. z_t acts as the forget gate and (1 - z_t) acts as the input gate, linked by the complementary constraint.

Practical Differences

• LSTM: Can have f_t = 0.9 and i_t = 0.9 simultaneously, keeping 90% old AND adding 90% new. Cell state magnitude can grow.
• GRU: If z_t = 0.9, then (1 - z_t) = 0.1. Keeping and replacing are a zero-sum game. Hidden state stays bounded.
• LSTM: Has an output gate o_t filtering cell state: h_t = o_t \odot \tanh(c_t). GRU exposes h_t directly.

The GRU trades flexibility (independent gates, separate cell/hidden states) for simplicity: 3 gate matrices instead of 4. Cho et al. (2014) and Chung et al. (2014) showed this tradeoff is often favorable, especially with limited training data.

Gradient Flow

Derivative Through the Interpolation

\frac{\partial h_t}{\partial h_{t-1}} = \text{diag}(z_t) + \text{diag}(1 - z_t) \cdot \frac{\partial \tilde{h}_t}{\partial h_{t-1}} + \text{terms from } \frac{\partial z_t}{\partial h_{t-1}}

The first term, \text{diag}(z_t), provides a direct additive path from h_{t-1} to h_t. When z_t^{(k)} \approx 1, the gradient for dimension k passes through with near-unit magnitude.

The Gradient Highway

For a dimension where z_\tau^{(k)} \approx 1 across time steps t+1 through T:

\prod_{\tau=t+1}^{T} z_\tau^{(k)} \approx 1

Unlike vanilla RNNs where gradients pass through tanh at every step (exponential decay), the GRU provides a linear shortcut. Gradients traverse many time steps without vanishing, as long as the update gate keeps those dimensions open.

Two Gradient Paths

• Direct path (z_t \odot h_{t-1}): Gradient scaled by z_t. The "memory highway" preventing vanishing gradients.
• Candidate path ((1 - z_t) \odot \tilde{h}_t): Gradient flows through tanh and the reset gate. Subject to attenuation, but necessary for learning new representations.

Both paths contribute simultaneously. The direct path ensures learning signals reach earlier time steps even when the candidate path's gradient is small, analogous to residual connections in feedforward networks.

Pitfalls

Swapping z_t and (1 - z_t)

Writing h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t reverses the semantics: z = 1 would mean "take the candidate" instead of "keep old state." The correct formula has z_t multiplying h_{t-1}. Mnemonic: z is the "lazy gate" -- high z means the unit does not bother updating.

Using Matrix Multiplication Instead of Element-wise

The \odot symbol means element-wise multiplication. Both z_t and h_{t-1} have shape (N, H), and the result has shape (N, H). Using np.matmul or @ instead of * produces shape errors or silently wrong results.

Confusing This Step with the Candidate Computation

The candidate computation uses the reset gate r_t and applies tanh. The hidden state update uses the update gate z_t and has no nonlinearity. Students sometimes apply tanh to the final output or use r_t where z_t belongs.

Forgetting the Complement

Computing h_t = z_t \odot h_{t-1} + z_t \odot \tilde{h}_t (using z_t for both terms) breaks the convex combination. Coefficients no longer sum to one, and hidden state magnitude drifts. The (1 - z_t) complement is essential.

Assuming z = 1 Means "Update"

The name "update gate" is misleading. When z_t = 1: h_t = 1 \cdot h_{t-1} + 0 \cdot \tilde{h}_t = h_{t-1}. The unit does not update. When z_t = 0: h_t = 0 \cdot h_{t-1} + 1 \cdot \tilde{h}_t = \tilde{h}_t. Full update. The gate value represents how much to keep, not how much to change.

Ignoring Batch Dimensions

All tensors have shape (N, H) where N is the batch size. The interpolation runs independently per sample. Incorrect reshaping can cause cross-sample contamination. Element-wise operations broadcast correctly across the batch dimension without special handling.

Examples

Hints

1. Compute z_t * h_prev + (1.0 - z_t) * h_tilde elementwise.

Requirements

• Blend the previous and candidate states elementwise using z_t.
• Return a float64 NumPy array with the same shape as the inputs.

Constraints

• All three arrays have the same shape, either (H,) or (N,H).
• Gate values lie in [0,1].
• Every input is float64 and finite.

Starter Code

import numpy as np

def hidden_state_update(z_t: np.ndarray, h_prev: np.ndarray,
                        h_tilde: np.ndarray) -> np.ndarray:
    """
    Returns the float64 updated hidden state.
    """
    pass

Test Cases