ReLU Activation Function

ImageNet Classification with Deep Convolutional Neural Networks

Easy

Problem

Theory

The Rectified Linear Unit (ReLU) is defined as f(x) = \max(0, x). In the 2012 paper "ImageNet Classification with Deep Convolutional Neural Networks," Krizhevsky, Sutskever, and Hinton showed that replacing sigmoid/tanh with ReLU allowed AlexNet to train several times faster while achieving record-breaking performance on ILSVRC-2012.

What It Is / What It Does

An activation function injects non-linearity into a neural network. Without it, stacked linear layers collapse into a single linear model regardless of depth.

ReLU does this with extreme simplicity:

• Positive inputs pass through unchanged
• Negative inputs are set to zero
• No learned parameters and no expensive computations (no exponentials, no divisions)
• Element-wise operation with no interaction between elements

Each neuron operates linearly, but the network creates a piecewise linear approximation of complex functions. A deep ReLU network can produce an exponential number of linear regions.

In AlexNet, ReLU is applied after every convolutional and fully connected layer, independently to all scalar values in each feature map.

Key Equations

ReLU Definition

f(x) = \max(0, x) = \begin{cases} x & \text{if } x > 0 \\ 0 & \text{if } x \leq 0 \end{cases}

ReLU Derivative (Gradient)

f'(x) = \begin{cases} 1 & \text{if } x > 0 \\ 0 & \text{if } x < 0 \end{cases}

f'(0) is technically undefined. In practice, implementations assign f'(0) = 0 with no measurable effect on training.

Sigmoid Function (for comparison)

\sigma(x) = \frac{1}{1 + e^{-x}}

Derivative: \sigma'(x) = \sigma(x)(1 - \sigma(x)), max 0.25 at x = 0. Gradient always attenuated by at least 4x per layer, compounding through depth.

Tanh Function (for comparison)

\tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}

Derivative: \tanh'(x) = 1 - \tanh^2(x), max 1.0 at x = 0, but < 1 for all x \neq 0 and decays rapidly for |x| > 2. Both sigmoid and tanh are saturating nonlinearities. ReLU is non-saturating for positive inputs: gradient is always exactly 1.

Mechanics / How It Works

Element-wise Operation

Given input tensor \mathbf{Z}, output \mathbf{A} has the same shape with a_{ij} = \max(0, z_{ij}). No interactions between elements, no parameters, no state.

Behavior on Positive Values

When z > 0, ReLU acts as identity: f(z) = z, gradient exactly 1. The upstream gradient passes through unmodified during backpropagation. This is the key property: gradient signal travels backward through many layers without attenuation, as long as neurons are active.

Behavior on Negative Values

When z \leq 0, ReLU outputs zero and gradient is zero. This creates dynamic sparsity: for any given input, only a subset of neurons are active. Krizhevsky et al. noted this sparsity is computationally beneficial, as the network effectively uses a different sparse sub-network for each input.

Gradient Flow During Backpropagation

\frac{\partial L}{\partial z} = \frac{\partial L}{\partial a} \cdot f'(z) = \begin{cases} \frac{\partial L}{\partial a} & \text{if } z > 0 \\ 0 & \text{if } z \leq 0 \end{cases}

ReLU during backpropagation acts as a binary mask: gradients pass where neurons were active, blocked where inactive. No attenuation for active neurons. With sigmoid, every layer multiplies by \leq 0.25; after 10 layers: $ \approx 9.5 \times 10^{-7}$. With ReLU, active paths have multiplicative factors of exactly 1.

Paper Context / Design Decisions

Why Krizhevsky et al. Chose ReLU

In Section 3.1 ("ReLU Nonlinearity"), the authors state: "Deep convolutional neural networks with ReLUs train several times faster than their equivalents with tanh units." Their experiment: a four-layer CNN on CIFAR-10 with ReLU reached 25% training error six times faster than tanh (Figure 1). They clarify: "This is not a regularization effect, because ReLUs do not saturate."

Enabling the Depth of AlexNet

AlexNet has 5 convolutional + 3 fully connected layers (8 learned layers, ~60M parameters). The paper credits ReLU as a key enabler: without fast ReLU training, they could not have trained this architecture on ImageNet (1.2M images, 1,000 classes) in practical time on two GTX 580 GPUs.

The authors cite prior work by Jarrett et al. (2009) and Nair and Hinton (2010), but AlexNet was the first to validate ReLU at scale on a large, challenging dataset, making it the de facto standard.

The Broader Architectural Impact

ReLU interacted with other AlexNet design choices. Its sparsity complemented dropout (both encourage distributed representations). Its unbounded positive output made local response normalization (LRN) useful for preventing dominant activations among neighboring neurons.

The Vanishing Gradient Problem

Why Sigmoid and Tanh Suffer

In a deep network with L layers, the gradient of the loss w.r.t. weights in layer l involves a product through all subsequent layers:

\frac{\partial L}{\partial \mathbf{W}^{(l)}} = \frac{\partial L}{\partial \mathbf{a}^{(L)}} \cdot \prod_{k=l+1}^{L} \text{diag}(\sigma'(\mathbf{z}^{(k)})) \cdot \mathbf{W}^{(k)} \cdot \frac{\partial \mathbf{a}^{(l)}}{\partial \mathbf{W}^{(l)}}

For sigmoid, \sigma'(x) \leq 0.25. After 8 layers: $ \approx 1.5 \times 10^{-5}$, so early-layer gradients are five orders of magnitude smaller than at the output.

For tanh, \tanh'(0) = 1 but drops rapidly: \tanh'(1) \approx 0.42, \tanh'(2) \approx 0.07, \tanh'(3) \approx 0.01. Unless all pre-activations cluster near zero (unrealistic), gradients still vanish exponentially.

How ReLU Solves This

For active neurons (z > 0), f'(z) = 1. The product along any path of active neurons:

\prod_{k=l+1}^{L} f'(z^{(k)}) = 1^{L-l} = 1

Zero attenuation from the activation function -- a qualitative change from sigmoid/tanh. Inactive neurons (z \leq 0) have zero gradient (the "dying ReLU" problem, see Pitfalls), but in a healthy network sufficient neurons remain active for gradient flow through alternative paths.

The Mathematical Argument

Under typical conditions, \mathbb{E}[|\sigma'(z)|] \approx 0.2 for sigmoid. For ReLU with ~50% active neurons, \mathbb{E}[f'(z)] \approx 0.5. Over 8 layers: $ / 0.2^8 = 3.9 \times 10^{-3} / 2.56 \times 10^{-6} \approx 1500\times$ more gradient reaching early layers. Moreover, ReLU gradients that pass through arrive at full strength (factor of 1), unlike sigmoid's always-diminished gradients.

Numerical Example

Setup

Pre-activation values from a single layer:

\mathbf{z} = [-2.0, \; -0.5, \; 0.0, \; 0.3, \; 1.5, \; -1.0, \; 2.7, \; 0.1]

Step 1: Forward Pass (Computing ReLU Output)

Apply f(x) = \max(0, x) element-wise:

• f(-2.0) = 0.0, f(-0.5) = 0.0, f(0.0) = 0.0 (negative/zero, zeroed out)
• f(0.3) = 0.3, f(1.5) = 1.5, f(2.7) = 2.7, f(0.1) = 0.1 (positive, pass through)
• f(-1.0) = 0.0 (negative, zeroed out)

ReLU output:

\mathbf{a} = [0.0, \; 0.0, \; 0.0, \; 0.3, \; 1.5, \; 0.0, \; 2.7, \; 0.1]

4 out of 8 values (50%) are zeroed out -- typical and desirable sparsity.

Step 2: Backward Pass (Computing ReLU Gradient)

Upstream gradient:

\frac{\partial L}{\partial \mathbf{a}} = [0.4, \; -0.2, \; 0.1, \; -0.6, \; 0.8, \; 0.3, \; -0.5, \; 0.9]

Apply binary mask (pass gradient where z > 0, block where z \leq 0):

• z_1=-2.0 \leq 0 \rightarrow 0.0; z_2=-0.5 \leq 0 \rightarrow 0.0; z_3=0.0 \leq 0 \rightarrow 0.0; z_6=-1.0 \leq 0 \rightarrow 0.0
• z_4=0.3 > 0 \rightarrow -0.6; z_5=1.5 > 0 \rightarrow 0.8; z_7=2.7 > 0 \rightarrow -0.5; z_8=0.1 > 0 \rightarrow 0.9

Gradient passed to previous layer:

\frac{\partial L}{\partial \mathbf{z}} = [0.0, \; 0.0, \; 0.0, \; -0.6, \; 0.8, \; 0.0, \; -0.5, \; 0.9]

Every active neuron passes its gradient at full magnitude -- no attenuation.

Step 3: Comparison with Sigmoid

Sigmoid gradient factors \sigma'(z) = \sigma(z)(1-\sigma(z)) for the same inputs:

• z=-2.0: 0.105; z=-0.5: 0.235; z=0.0: 0.250; z=0.3: 0.245
• z=1.5: 0.149; z=-1.0: 0.197; z=2.7: 0.059; z=0.1: 0.249

Even the best case (z=0) attenuates to 25%. At $$, gradient is reduced to 5.9%. These attenuations compound catastrophically over multiple layers.

Step 4: Comparison with Tanh

Tanh gradient factors 1 - \tanh^2(z):

• z=-2.0: 0.071; z=-0.5: 0.786; z=0.0: 1.000; z=0.3: 0.915
• z=1.5: 0.181; z=-1.0: 0.420; z=2.7: 0.020; z=0.1: 0.990

Tanh reaches 1.0 at z=0 but drops to 0.020 at $$. ReLU gives exactly 1.0 for all positive inputs regardless of magnitude: $ = f'(0.001) = 1$.

Variants and Modern Context

Leaky ReLU

f(x) = \begin{cases} x & \text{if } x > 0 \\ \alpha x & \text{if } x \leq 0 \end{cases}

Typically \alpha = 0.01. Ensures every neuron always has a non-zero gradient, preventing complete "death." Proposed by Maas et al. (2013), popular in GANs.

Parametric ReLU (PReLU)

f(x) = \begin{cases} x & \text{if } x > 0 \\ \alpha_i x & \text{if } x \leq 0 \end{cases}

He et al. (2015) made \alpha_i a learnable per-channel parameter. Combined with He/Kaiming initialization (which accounts for ReLU's variance properties), PReLU achieved superhuman ImageNet performance. Learned \alpha values typically converge to small positive numbers.

Exponential Linear Unit (ELU)

f(x) = \begin{cases} x & \text{if } x > 0 \\ \alpha(e^x - 1) & \text{if } x \leq 0 \end{cases}

Clevert et al. (2016), typically \alpha = 1.0. Produces negative outputs (pushing mean closer to zero), saturates to -\alpha for large negatives. More expensive than ReLU due to exponential computation; practical benefits often marginal.

GELU (Gaussian Error Linear Unit)

f(x) = x \cdot \Phi(x) = x \cdot \frac{1}{2}\left[1 + \text{erf}\left(\frac{x}{\sqrt{2}}\right)\right]

Hendrycks and Gimpel (2016). A smooth, probabilistic version of ReLU: scales each input by the probability a standard normal variable would be less than that input. Used in BERT, GPT-2, GPT-3, and most modern transformers.

Swish / SiLU (Sigmoid Linear Unit)

f(x) = x \cdot \sigma(x) = \frac{x}{1 + e^{-x}}

Ramachandran et al. (2017), Google Brain. Smooth, non-monotonic, unbounded above. Consistently outperformed ReLU across architectures. Used in EfficientNet. Closely related to GELU in practice.

Why ReLU Remains Dominant

• Simplicity: trivial to implement, no hyperparameters
• Efficiency: single comparison per element vs. exponentials/error functions
• Well-understood initialization: He initialization is specifically designed and validated for ReLU
• Robustness: rarely fails catastrophically, strong performance across tasks and architectures

Pitfalls

The Dying ReLU Problem

A neuron "dies" when its weights evolve such that z = \mathbf{w}^T \mathbf{x} + b < 0 for all training inputs. Output and gradient are permanently zero -- the neuron never recovers. Most likely with high learning rates or negative bias initialization. Mitigation: zero/small-positive bias init, moderate learning rates, or use Leaky ReLU/PReLU/ELU.

Non-Zero-Centered Outputs

ReLU outputs are always \geq 0, so all inputs to the next layer are positive. This forces all weight gradients for that neuron to share the same sign, constraining optimization to zig-zag paths in weight space. Batch normalization largely mitigates this by re-centering activations.

Non-Differentiability at Zero

ReLU has a kink at x = 0. In practice this is a non-issue: the probability of exactly 0.0 in floating-point is negligible, and most implementations set f'(0) = 0. ReLU networks converge under standard conditions despite this.

Unbounded Activations

Unlike sigmoid (0 to 1) or tanh (-1 to 1), ReLU has no upper bound. Activations can grow large, risking numerical overflow and unstable gradients. Batch normalization, layer normalization, and weight decay are important companions. In AlexNet, LRN served this role.

Effect on Batch Statistics

ReLU zeros ~50% of inputs, shifting mean upward and reducing variance. He initialization compensates by scaling weights by \sqrt{2/n} instead of \sqrt{1/n}. If batch norm is applied before ReLU, the normalized distribution is truncated to a half-normal (mean \approx 0.4, variance \approx 0.68). The ordering of batch norm and ReLU remains a subject of practical consideration.

Sensitivity to Weight Initialization

ReLU behavior depends on the sign of pre-activations, making initialization critical. He initialization (variance 2/n) keeps activation variance constant across layers. Xavier/Glorot initialization (variance 1/n), designed for sigmoid/tanh, causes variance to shrink by 2x at each ReLU layer, leading to vanishing activations in deep networks. Wrong initialization can cause complete training failure.

Examples

Hints

1. np.maximum can compare the whole array with zero elementwise.

Requirements

• Replace every negative element with zero.
• Keep zero and positive elements unchanged.
• Return a new float64 NumPy array with the same shape as x.

Constraints

• x is a nonempty float64 NumPy array of any rank.
• Every input value is finite.

Starter Code

import numpy as np

def relu(x: np.ndarray) -> np.ndarray:
    """
    Returns the elementwise float64 ReLU output.
    """
    pass

Test Cases