SELU Activation

Activation Functions

Easy

Problem

Theory

Deep networks suffer from internal covariate shift: the distribution of activations at each layer changes as the network trains. This makes training unstable because each layer must constantly adapt to the shifting statistics of its inputs.

The standard fix is batch normalization: normalize each layer's activations to have zero mean and unit variance, then learn a scale and shift. It works well, but:

• It depends on batch statistics, so behavior differs between training and inference
• It does not work well with small batch sizes
• It adds computational overhead and extra parameters
• It complicates the model architecture

What if the activation function itself could maintain stable statistics automatically?

SELU: Self-Normalizing Activations

SELU (Scaled Exponential Linear Unit) was designed to do exactly this. It is a scaled version of ELU with specific constants chosen so that activations automatically converge to zero mean and unit variance as they pass through layers.

\text{SELU}(x) = \lambda \cdot \begin{cases} x & \text{if } x > 0 \\ \alpha \cdot (e^x - 1) & \text{if } x \leq 0 \end{cases}

The constants are not arbitrary. They were derived analytically:

\lambda \approx 1.0507 \qquad \alpha \approx 1.6733

These exact values are what make the self-normalizing property work. You cannot change them without breaking the mathematical guarantee.

How Self-Normalization Works

The key insight (from the paper by Klambauer et al., 2017): if activations enter a SELU layer with mean 0 and variance 1, they exit with mean 0 and variance 1. This holds approximately even after the nonlinearity.

Why these specific constants?

• \lambda > 1: the scale factor is slightly greater than 1. This means values that are too small get amplified, pulling the variance back up toward 1.
• \alpha \approx 1.6733: the ELU's negative saturation level. This is chosen so that the negative outputs produce just enough mean shift to counteract the positive side's bias.
• Together, the constants create a fixed-point attractor: if the mean and variance drift away from (0, 1), the SELU function pulls them back.

The mathematical proof shows that there is a unique fixed point at mean 0 and variance 1, and that SELU converges toward this fixed point. This is why it is called "self-normalizing."

Some Concrete Values

• \text{SELU}(1.0) = \lambda \cdot 1.0 \approx 1.0507
• \text{SELU}(0) = \lambda \cdot \alpha \cdot (e^0 - 1) = 0
• \text{SELU}(-1.0) = \lambda \cdot \alpha \cdot (e^{-1} - 1) \approx 1.0507 \times 1.6733 \times (-0.632) \approx -1.111
• \text{SELU}(-5.0) \approx 1.0507 \times 1.6733 \times (-0.993) \approx -1.746
• As x \to -\infty: \text{SELU}(x) \to -\lambda \cdot \alpha \approx -1.758

Notice the output range:

• Positive side: unbounded, scaled by \lambda \approx 1.05
• Negative side: saturates at approximately -1.758

The Requirements for Self-Normalization

SELU's self-normalizing property only works under specific conditions:

1. Weight initialization: must use LeCun normal initialization (weights drawn from N(0, 1/n) where n is the number of inputs). Other initializations break the property.
2. Architecture: works for fully connected (dense) layers. The proof does not directly apply to convolutional or recurrent layers.
3. Dropout variant: standard dropout breaks self-normalization. Use alpha dropout instead, which randomly sets activations to the negative saturation value -\lambda\alpha rather than zero.
4. Input normalization: the inputs to the network should be standardized (mean 0, variance 1).

When all conditions are met, SELU networks can be trained to significant depth (100+ layers) without batch normalization, and the activations maintain stable statistics throughout.

SELU vs. ELU

SELU is literally ELU with a specific scale factor:

\text{SELU}(x) = \lambda \cdot \text{ELU}(x, \alpha = 1.6733)

The differences:

• ELU: \alpha is a free hyperparameter. No self-normalizing guarantee.
• SELU: \alpha and \lambda are fixed constants. Self-normalizing when conditions are met.
• ELU: works with any architecture and initialization
• SELU: requires LeCun initialization and fully connected layers for the guarantee

Where SELU Shows Up

• Deep fully connected networks: this is where SELU shines. Networks with 50+ dense layers can train without batch normalization.
• Tabular data: SELU is popular for non-image, non-text tasks where the architecture is a deep MLP
• Autoencoders: SELU works well in deep autoencoder architectures
• As a baseline: SELU is useful as a "does this task even need batch normalization?" test. If SELU performs comparably, you can skip the complexity of batch norm.

Examples

Hints

1. Store the fixed lambda and alpha constants inside the function.
2. Round each transformed value to four decimal places before appending it.

Requirements

• Apply the SELU formula element-wise
• Use the exact constants: lambda = 1.0507009873554804934193349852946, alpha = 1.6732632423543772848170429916717
• Return a list of floats with the same length as input

Constraints

• Input list has at least one element
• Use the provided constant values for lambda and alpha
• Return a list of floats with the same length as input
• Time limit: 300 ms

Starter Code

import math

def selu(x: list) -> list:
    """
    Returns SELU values rounded to four decimal places.
    """
    # Write code here
    pass

Test Cases