ELU Activation

Activation Functions

Easy

Problem

Theory

ReLU (\max(0, x)) revolutionized deep learning, but it has two issues:

1. Dead neurons: when the input is negative, the output and gradient are both zero. Neurons can die permanently and never recover.
2. Non-zero mean outputs: ReLU only outputs zero or positive values, so its average output is always positive. This introduces a bias shift that can slow down learning in deeper layers.

Leaky ReLU fixes the dead neuron problem by allowing a small slope for negative inputs (\alpha x). But Leaky ReLU's negative side is just a straight line, which does not address the mean shift issue.

ELU: Exponential Curve for Negatives

ELU (Exponential Linear Unit) takes a different approach to the negative side. Instead of a straight line, it uses an exponential curve that smoothly saturates at -\alpha:

\text{ELU}(x) = \begin{cases} x & \text{if } x > 0 \\ \alpha \cdot (e^x - 1) & \text{if } x \leq 0 \end{cases}

Some values to build intuition (with \alpha = 1.0):

• \text{ELU}(2.0) = 2.0 (positive side: identity)
• \text{ELU}(0) = 1.0 \cdot (e^0 - 1) = 0 (continuous at zero)
• \text{ELU}(-0.5) = 1.0 \cdot (e^{-0.5} - 1) \approx -0.394
• \text{ELU}(-1.0) = 1.0 \cdot (e^{-1} - 1) \approx -0.632
• \text{ELU}(-5.0) = 1.0 \cdot (e^{-5} - 1) \approx -0.993
• \text{ELU}(-\infty) \to -\alpha = -1.0

The negative side curves smoothly from 0 toward -\alpha and stays there.

Why the Exponential Shape Helps

The exponential curve has three benefits over both ReLU and Leaky ReLU:

1. Mean activations closer to zero

Since ELU outputs negative values that saturate at -\alpha, the average output across a layer is closer to zero than with ReLU. This is important because:

• Layers with zero-mean activations converge faster
• The bias shift from always-positive outputs in ReLU acts like an unwanted bias term that the next layer must compensate for
• Closer-to-zero mean activations have a similar effect to batch normalization, but built into the activation function itself

2. No dead neurons

For any negative input x, the gradient is:

\frac{d}{dx} \text{ELU}(x) = \alpha \cdot e^x

This is always positive (never zero), so gradients always flow. Unlike ReLU, neurons cannot die.

3. Smooth transition at zero

ELU is continuous and differentiable everywhere, including at x = 0. There is no sharp corner like ReLU has. The smooth curve means:

• The optimization landscape has fewer sharp edges
• Gradient-based optimization is more stable
• The derivative transitions smoothly from the exponential decay to 1

The Alpha Parameter

\alpha controls the negative saturation value:

• \alpha = 1.0: the most common choice. The output saturates at -1, which pairs well with the positive side's slope of 1.
• Smaller \alpha (e.g., 0.5): less negative output, weaker regularization effect
• Larger \alpha (e.g., 2.0): more negative output, stronger mean-pushing effect

As x \to -\infty, ELU approaches -\alpha. So \alpha directly sets the floor for the negative output.

ELU vs. ReLU vs. Leaky ReLU

At x = -1 (with \alpha = 1.0 for ELU, \alpha = 0.01 for Leaky ReLU):

• ReLU: output = 0, gradient = $$
• Leaky ReLU: output = -0.01, gradient = $$
• ELU: output \approx -0.632, gradient \approx 0.368

Key differences:

• ReLU is the fastest to compute (just a comparison) but risks dead neurons
• Leaky ReLU prevents dead neurons with minimal computation overhead, but does not help with mean shift
• ELU prevents dead neurons and pushes mean activations toward zero, but requires computing e^x which is slower

Where ELU Shows Up

• Deep fully connected networks: ELU often outperforms ReLU when batch normalization is not used, because it provides implicit mean-centering
• As a building block for SELU: the Scaled ELU (SELU) multiplies ELU by a specific constant to achieve self-normalizing properties
• Image recognition tasks: papers have shown ELU competitive with or better than ReLU on CIFAR and ImageNet benchmarks
• Wherever you need smooth gradients: the differentiability at zero makes ELU useful when gradient smoothness matters

Examples

Hints

1. Use the original value when it is positive.
2. Use alpha times exp(value) minus one for the nonpositive branch.

Requirements

• Apply the ELU formula element-wise to each value in the input list
• Use the alpha parameter for negative values
• Return a list of floats with the same length as the input

Constraints

• Input list has at least one element
• alpha >= 0
• Return a list of floats with the same length as input
• Time limit: 300 ms

Starter Code

import math

def elu(x: list, alpha: float = 1.0) -> list:
    """
    Returns ELU applied elementwise to the input values.
    """
    # Write code here
    pass

Test Cases