Data Augmentation

ImageNet Classification with Deep Convolutional Neural Networks

Medium

Problem

Theory

Data augmentation artificially expands a training dataset by applying label-preserving transformations to existing images. In AlexNet (Krizhevsky, Sutskever & Hinton, 2012), augmentation was a primary defense against overfitting a 60-million-parameter network on ~1.2 million ImageNet images. The authors used two augmentation strategies, both computed on CPU while the GPU trained on the previous batch.

What Data Augmentation Is and What It Does

Augmentation takes an original training example and produces new examples via random transformations that preserve the semantic label. A photo of a cat remains a cat whether cropped, flipped, or color-shifted.

• Expanding the training set: If each image yields k augmented versions, the effective dataset grows by factor k. AlexNet's crop-and-flip scheme produces 2048 unique 224x224 patches per 256x256 image.
• Reducing overfitting: The network almost never sees the exact same pixel arrangement twice, forcing it to learn robust features rather than memorize specific patterns.
• Improving generalization: Augmentation encodes prior knowledge about invariances (translation, reflection, illumination) directly into the data pipeline.
• Critical for deep networks: AlexNet had ~60M parameters but only ~1.2M training images. Without augmentation and dropout, the network would overfit severely.

Key Equations

Random Crop Extraction

Given image I of size 256 \times 256, a 224 \times 224 crop is extracted at random position (y, x):

I_{\text{crop}} = I[y : y + h, \; x : x + w, \; :]

where offsets are sampled uniformly:

y \sim \text{Uniform}\{0, 1, \ldots, H - h\}, \quad x \sim \text{Uniform}\{0, 1, \ldots, W - w\}

Both y and x range from 0 to 32, giving (256 - 224)^2 = 1024 distinct positions (the paper's rounding).

Horizontal Flip

A horizontal flip reverses the width axis, applied with probability 0.5:

I_{\text{flip}}[y, x, c] = I_{\text{crop}}[y, \; w - 1 - x, \; c]

I_{\text{aug}} = \begin{cases} I_{\text{flip}} & \text{with probability } 0.5 \\ I_{\text{crop}} & \text{with probability } 0.5 \end{cases}

PCA Color Augmentation

Let \mathbf{p}_1, \mathbf{p}_2, \mathbf{p}_3 \in \mathbb{R}^3 be eigenvectors and \lambda_1, \lambda_2, \lambda_3 eigenvalues of the 3 \times 3 RGB covariance matrix over the training set. The perturbation added to every pixel:

\Delta I = [\mathbf{p}_1, \mathbf{p}_2, \mathbf{p}_3] \begin{bmatrix} \alpha_1 \lambda_1 \\ \alpha_2 \lambda_2 \\ \alpha_3 \lambda_3 \end{bmatrix}

where each \alpha_i \sim \mathcal{N}(0, 0.1). The same \Delta I \in \mathbb{R}^3 is added to every pixel, producing a coherent color shift rather than per-pixel noise.

AlexNet's Two Augmentation Strategies

Strategy 1: Random Cropping with Horizontal Flipping

• Rescale: Resize so shorter side is 256px, then take central 256x256 crop.
• Random crop: Extract a 224x224 patch from a random position, with top-left (y, x) sampled from [0, 32].
• Random flip: Flip horizontally with probability 0.5, doubling the effective patches.
• Augmentation factor: 1024 \times 2 = 2048 distinct augmented examples per image.

Cropping introduces translation invariance; flipping introduces mirror-reflection invariance. Both are appropriate for most ImageNet object categories.

Strategy 2: PCA Color Augmentation (Fancy PCA)

PCA is performed on all RGB pixel values across ImageNet, yielding three eigenvectors and eigenvalues capturing principal color variation. At training time, three scalars \alpha_i \sim \mathcal{N}(0, 0.1) are sampled per image:

\Delta I = \alpha_1 \lambda_1 \mathbf{p}_1 + \alpha_2 \lambda_2 \mathbf{p}_2 + \alpha_3 \lambda_3 \mathbf{p}_3

The first principal component roughly corresponds to brightness; the others capture color-opponent variation. The paper notes: "This scheme approximately captures an important property of natural images, namely, that object identity is invariant to changes in the intensity and color of the illumination."

Fresh \alpha_i values are drawn once per image per epoch, ensuring continued diversity without storing augmented images.

Paper Context and Design Decisions

Computational Efficiency

Both strategies generate transformed images on-the-fly from originals with minimal computation, avoiding disk storage. The paper states: "The image transformations are generated in Python code on the CPU while the GPU is training on the previous batch of images. So these data augmentation schemes are, in effect, computationally free."

Error Reduction

PCA color augmentation reduced top-1 error by over 1%. On ImageNet LSVRC, where competition margins were often under 1%, this was significant. The crop-and-flip contribution is harder to isolate but was essential to prevent severe overfitting.

On-the-Fly Generation

Augmentation is applied during training, not as preprocessing. Each epoch, the network sees a different augmented version of each image. This on-the-fly approach became standard in all subsequent deep learning pipelines.

Why Augmentation Works as Regularization

• Never sees the same image twice: With 1024 crop positions, 2 flip states, and continuous color perturbation, the space of augmented versions per image is effectively infinite. This prevents memorization.
• Dataset size multiplication: Multiplying effective dataset size by 2048 (discrete augmentations alone) reduces variance in the bias-variance tradeoff, pushing the model toward features capturing the true distribution.
• Forcing invariance: Training on flipped images teaches flip invariance; translated crops teach translation invariance; color perturbation teaches illumination invariance. Each reduces the effective complexity of the function to learn.
• Bias-variance connection: Augmentation biases the network toward transformation-invariant solutions. Since the true data-generating process is indeed invariant to these transformations, this bias is beneficial, yielding reduced variance with negligible increase in bias.
• Vicinal Risk Minimization (VRM): Standard ERM only considers exact training examples. Augmentation implements VRM by minimizing risk over a "vicinity" of each example, defined by the set of plausible transformations.

Test-Time Augmentation in AlexNet

The 10-Crop Procedure

At inference, each test image (256x256) yields 10 fixed crops of 224x224:

• Four corner crops: top-left, top-right, bottom-left, bottom-right.
• One center crop.
• Five flipped versions: horizontal flip of each above crop.

The final prediction averages the 10 softmax outputs:

\hat{y} = \frac{1}{10} \sum_{i=1}^{10} f(I_i)

Why It Works

Each crop provides a different view; averaging reduces prediction variance. For uncorrelated predictions with variance \sigma^2, averaging yields variance \sigma^2 / 10. In practice, crops are correlated (overlapping regions), so the reduction is less, but still significant. Not using this procedure and relying on center crop alone led to noticeably worse results.

This was one of the earliest systematic TTA applications in deep learning. Modern TTA generalizes this with random augmentations and many more views; some competition solutions use hundreds of augmented views per test image.

Numerical Example

A concrete walkthrough of all augmentation operations on a single training image.

Step 1: Start with a 256x256 Image

Training image I of shape (256, 256, 3): a golden retriever with RGB values in [0, 255].

Step 2: Random Crop

Sample y = 17, x = 24 (both in valid range [0, 32]):

I_{\text{crop}} = I[17:241, \; 24:248, \; :]

Result: a (224, 224, 3) patch, shifting the retriever slightly up-left relative to center.

Step 3: Random Horizontal Flip

Sample u = 0.37. Since u < 0.5, flip:

I_{\text{flip}}[y, x, c] = I_{\text{crop}}[y, \; 223 - x, \; c]

The retriever facing left now faces right. Label preserved.

Step 4: PCA Color Augmentation

Representative ImageNet eigenvectors and eigenvalues:

\mathbf{p}_1 = \begin{bmatrix} -0.5675 \\ -0.5808 \\ -0.5836 \end{bmatrix}, \quad \mathbf{p}_2 = \begin{bmatrix} -0.7192 \\ 0.0045 \\ 0.6948 \end{bmatrix}, \quad \mathbf{p}_3 = \begin{bmatrix} -0.4009 \\ 0.8140 \\ -0.4203 \end{bmatrix}

\lambda_1 = 0.2175, \quad \lambda_2 = 0.0188, \quad \lambda_3 = 0.0045

Note \lambda_1 \gg \lambda_2 \gg \lambda_3: most color variation is along the first component (roughly brightness). Sample \alpha_1 = 0.12, \alpha_2 = -0.05, \alpha_3 = 0.08 from \mathcal{N}(0, 0.1):

Term 1: \alpha_1 \lambda_1 \mathbf{p}_1 = 0.0261 \times [-0.5675, -0.5808, -0.5836]^T = [-0.0148, -0.0152, -0.0152]^T

Term 2: \alpha_2 \lambda_2 \mathbf{p}_2 = -0.00094 \times [-0.7192, 0.0045, 0.6948]^T = [0.000676, -0.0000042, -0.000653]^T

Term 3: \alpha_3 \lambda_3 \mathbf{p}_3 = 0.00036 \times [-0.4009, 0.8140, -0.4203]^T = [-0.000144, 0.000293, -0.000151]^T

Total:

\Delta I = \begin{bmatrix} -0.01427 \\ -0.01491 \\ -0.01600 \end{bmatrix}

All channels decrease similarly because the dominant perturbation is along \mathbf{p}_1 (roughly equal components), producing a slight darkening. In [0, 255] scale: \approx [-3.6, -3.8, -4.1], barely perceptible but meaningful over many epochs.

Step 5: Final Augmented Image

I_{\text{aug}}[y, x, :] = I_{\text{flip}}[y, x, :] + \Delta I

This is one of effectively infinite possible views of the original image.

Modern Augmentation Techniques

AlexNet's strategies were pioneering. The field has since expanded significantly.

• AutoAugment / RandAugment: AutoAugment (Cubuk et al., 2019) uses RL to search for optimal augmentation policies. RandAugment (Cubuk et al., 2020) simplifies this to two hyperparameters (N transformations, shared magnitude M), achieving comparable results at far less cost.
• CutMix: (Yun et al., 2019) Pastes a rectangular region from one image onto another, mixing labels proportionally to area. Goes beyond label-preserving into label-mixing augmentation.
• Mixup: (Zhang et al., 2018) Blends two images and labels via convex combination: \tilde{x} = \lambda x_i + (1-\lambda)x_j, \tilde{y} = \lambda y_i + (1-\lambda)y_j, where \lambda \sim \text{Beta}(\alpha, \alpha).
• CutOut: (DeVries & Taylor, 2017) Randomly masks a square region with zeros, forcing robustness to partial occlusion. A spatial analog of dropout on the input.
• ColorJitter: PyTorch's torchvision.transforms.ColorJitter is the modern descendant of AlexNet's PCA augmentation, adjusting brightness, contrast, saturation, and hue with explicit control.

The evolution: expanded transformation sets, automated policy search, multi-image combinations (Mixup, CutMix), and feature-space augmentation. The core principle AlexNet demonstrated remains unchanged.

Pitfalls

• Augmentation at test time by mistake: Accidentally leaving random augmentation on during evaluation produces noisy, misleading metrics. Use deterministic preprocessing at test time (center crop, no flip, no color perturbation) unless doing deliberate TTA with averaged predictions.
• Augmentations that change the label: Not all transforms are label-preserving for all tasks. Flipping digit 6 vertically produces 9. Flipping a chest X-ray swaps left/right lungs. Augmentations must be chosen with domain knowledge.
• Color augmentation on color-discriminative tasks: PCA color augmentation suits ImageNet but harms tasks where color is discriminative (ripe vs. unripe tomatoes, satellite imagery band information).
• Wrong validation preprocessing: The validation set should use the same deterministic preprocessing as the test set (resize, center crop, normalize), not random augmentation. Reusing the training data loader for validation is a common source of this bug.
• Overly aggressive augmentation: Extreme transformations produce images that no longer resemble the real distribution. The network wastes capacity on unrealistic inputs. The right intensity is task-dependent.
• Forgetting to normalize after augmentation: Normalization (mean subtraction, std division) should be the last step. Correct order: spatial augmentations, then color augmentations, then clamping to valid range, then tensor conversion, then normalization.

Examples

Hints

1. A crop is image[crop_y:crop_y + crop_size, crop_x:crop_x + crop_size, :].
2. Horizontal flipping reverses axis 1 in HWC layout.
3. Call .copy() so the result does not share storage with the input.

Requirements

• random_crop must return the crop beginning at (crop_y, crop_x).
• random_horizontal_flip must reverse only the width axis when flip_rand is less than p.
• Neither function may modify the input array.
• Return a new float64 NumPy array.

Constraints

• image has shape (H,W,C).
• crop_size is positive and the requested crop lies inside the image.
• p and flip_rand are between 0 and 1, inclusive.
• Inputs and outputs use float64.

Starter Code

import numpy as np

def random_crop(image: np.ndarray, crop_size: int,
                crop_y: int, crop_x: int) -> np.ndarray:
    """
    Returns the float64 crop at the supplied coordinates.
    """
    pass

def random_horizontal_flip(image: np.ndarray, p: float,
                           flip_rand: float) -> np.ndarray:
    """
    Returns a new image, flipped when flip_rand is less than p.
    """
    pass

Test Cases