VGG Feature Extractor

Very Deep Convolutional Networks

Medium

Problem

Theory

The feature extractor is the convolutional backbone of VGGNet (Simonyan & Zisserman, 2014) that processes raw images into rich feature maps. It is defined entirely by a configuration list: integers specify convolutional layers (each followed by ReLU), and the character 'M' specifies a 2 \times 2 max pooling layer. This config-driven design was one of VGGNet's most influential contributions, enabling a single codebase to express VGG-11 through VGG-19.

What It Is

The feature extractor is the front half of a VGGNet model. It takes an input image tensor and passes it through a sequence of convolutional and pooling layers, producing a spatially downsampled tensor with many learned feature channels. The classifier (fully connected layers) sits after the feature extractor, but the feature extractor itself is where all spatial pattern detection happens.

Simonyan and Zisserman's key insight was that the entire convolutional backbone could be described by a flat list. Each element corresponds to exactly one layer: an integer means "create a Conv layer with this many output channels, followed by ReLU", and the string 'M' means "insert a 2 \times 2 max pooling layer with stride 2". The feature extractor iterates through this list from left to right, consuming weight-bias pairs for each convolutional layer and applying parameter-free pooling for each 'M'.

This separation of architecture specification (the config list) from architecture execution (the iteration loop) was a departure from prior work like AlexNet, where each layer was manually coded. It made systematic exploration of depth straightforward: the authors tested configurations A through E (11 to 19 layers) by changing only the config list.

Key Equations

The feature extractor applies two types of operations based on the config entry:

For an integer entry c_i (convolutional layer):

y = \text{ReLU}(x \cdot W_i + b_i)

where x is the current activation tensor, W_i is the weight matrix for the i-th convolutional layer, b_i is the bias vector, and ReLU is the element-wise nonlinearity \text{ReLU}(z) = \max(0, z). The output channels of y equal the integer value c_i. The input channels of W_i must match the channel dimension of x.

For an 'M' entry (max pooling layer):

y_{b,i,j,c} = \max(x_{b,2i,2j,c},\; x_{b,2i+1,2j,c},\; x_{b,2i,2j+1,c},\; x_{b,2i+1,2j+1,c})

This takes the maximum over each non-overlapping 2 \times 2 spatial window, halving both height and width while preserving channels and batch size. Max pooling has no learnable parameters.

The full feature extractor is the sequential composition:

\text{features}(x) = f_L \circ f_{L-1} \circ \ldots \circ f_2 \circ f_1(x)

where each f_k is either a Conv+ReLU or a MaxPool, determined by the k-th config entry.

The Config-Driven Architecture

The VGG paper introduced six configurations labeled A through E. Each is a flat list that fully specifies the convolutional backbone. The feature extractor code is identical across all variants; only the config list changes.

VGG-16 (Configuration D), the most widely used variant:

[64, 64, \text{'M'}, 128, 128, \text{'M'}, 256, 256, 256, \text{'M'}, 512, 512, 512, \text{'M'}, 512, 512, 512, \text{'M'}]

This encodes 13 convolutional layers and 5 max pooling layers. Within each "block" (layers between consecutive 'M' entries), channels stay constant. At each pooling boundary, channels double (64 to 128 to 256 to 512) while spatial dimensions halve. The 512-channel cap reflects a practical limit where further doubling would be prohibitively expensive.

VGG-19 (Configuration E) extends the deepest blocks to four conv layers each:

[64, 64, \text{'M'}, 128, 128, \text{'M'}, 256, 256, 256, 256, \text{'M'}, 512, 512, 512, 512, \text{'M'}, 512, 512, 512, 512, \text{'M'}]

The feature extraction code never changes across variants. A single loop dispatches to conv+ReLU for integers and to max pooling for 'M'. This makes the architecture trivially extensible and was a significant influence on later frameworks like PyTorch's nn.Sequential.

Layer-by-Layer Processing

The feature extractor maintains two pieces of mutable state as it iterates: the running activation tensor (starting as the input image) and a weight index tracking which conv weight-bias pair to consume next.

1. Initialize: Set the output tensor to the input x. Set the weight index w = 0.

2. For each entry in the config list: If the entry is an integer, fetch W_w and b_w, compute x \cdot W_w + b_w, apply ReLU, store the result, and increment w by 1. If the entry is 'M', apply $ \times 2$ max pooling. The weight index does not change because pooling has no parameters.

3. Return the final output tensor after all config entries have been processed.

The weight index is the critical bookkeeping variable. It advances only for convolutional layers, not for pooling. If the config has 13 integers (as in VGG-16), then exactly 13 weight-bias pairs must be provided, consumed in order. The first integer uses the first pair, the second integer uses the second pair, regardless of how many 'M' entries appear between them.

Feature Hierarchy

As the input passes through successive layers, features become progressively more abstract. The paper states: "the convolutional layers extract increasingly complex features as depth increases." This hierarchy is fundamental to deep convolutional networks and especially pronounced in VGGNet.

• Block 1 (64 channels): Early conv layers detect low-level primitives like edges, corners, and color gradients. Each filter responds to a specific oriented edge or color transition within its 3 \times 3 receptive field.
• Block 2 (128 channels): These layers combine edge responses into textures and simple patterns: grids, stripes, or specific color combinations spanning a slightly larger region.
• Block 3 (256 channels): Mid-level features emerge: parts of objects like wheels, eyes, or windows. The receptive field is large enough to capture spatially coherent structures.
• Blocks 4-5 (512 channels): Deep layers detect high-level semantic concepts: entire faces, animal bodies, or scene layouts. A single spatial position "sees" a large portion of the original image through accumulated receptive field growth.

Zeiler and Fergus (2014) demonstrated this hierarchy through visualization. VGGNet's uniform 3 \times 3 filters make the progression clean: each additional conv layer adds exactly 2 pixels to the theoretical receptive field diameter.

Spatial Dimension Tracking

VGGNet uses 3 \times 3 convolutions with padding 1 and stride 1 throughout. This "same" padding means conv layers preserve spatial dimensions exactly. Only max pooling changes spatial size, halving both height and width.

For a 224 \times 224 input through VGG-16:

1. Input: 224 \times 224 \times 3

2. After Block 1 (conv64, conv64, pool): 224 \to 112 \times 112 \times 64

3. After Block 2 (conv128, conv128, pool): 112 \to 56 \times 56 \times 128

4. After Block 3 (conv256, conv256, conv256, pool): 56 \to 28 \times 28 \times 256

5. After Block 4 (conv512, conv512, conv512, pool): 28 \to 14 \times 14 \times 512

6. After Block 5 (conv512, conv512, conv512, pool): 14 \to 7 \times 7 \times 512

The final output is 7 \times 7 \times 512 = 25{,}088 values per image. Five pooling layers produce a total downsampling factor of 2^5 = 32, and 224 / 32 = 7. This output is flattened and fed to the classifier's fully connected layers.

This clean geometric progression follows from VGGNet's uniform design: same-padding convolutions preserve resolution, and 2 \times 2 pooling with stride 2 halves it. No complex stride arithmetic is needed, unlike AlexNet's stride-4 first layer.

Paper Context

Simonyan and Zisserman's 2014 paper, "Very Deep Convolutional Networks for Large-Scale Image Recognition," studied how network depth affects classification accuracy. The central finding was that pushing depth from 11 to 19 layers with a homogeneous architecture (only 3 \times 3 convolutions) consistently improved ImageNet performance, achieving 7.3% top-5 error on ILSVRC-2014.

The feature extractor is the core contribution. Prior architectures used larger kernels (11 \times 11, 7 \times 7) in early layers. Simonyan and Zisserman showed that stacking multiple 3 \times 3 layers achieves the same receptive field with fewer parameters and more nonlinearities. Two stacked 3 \times 3 layers match one 5 \times 5 layer but use 2 \times 9 = 18 weights versus 25. Three stacked 3 \times 3 layers match 7 \times 7 with 27 weights versus 49.

The paper tested configurations A through E on the same data. Configuration A (11 layers) achieved 10.4% top-5 error, while E (19 layers) achieved 7.5%. This monotonic improvement with depth motivated later work on even deeper networks (GoogLeNet, ResNet).

Numerical Example

Consider a minimal feature extractor with config [2, \text{'M'}] on a 1 \times 4 \times 4 \times 1 input (batch 1, spatial $ \times 4$, 1 channel).

Input tensor x:

x = \begin{pmatrix} 1.0 & 2.0 & 0.5 & -1.0 \\ 3.0 & -0.5 & 1.5 & 2.0 \\ 0.0 & 1.0 & -2.0 & 0.5 \\ 2.0 & 0.5 & 1.0 & -0.5 \end{pmatrix}

Config entry 1: integer 2 (Conv with 2 output channels). Weight W_0 has shape (1, 2), bias b_0 has shape (2,).

Let W_0 = \begin{pmatrix} 0.5 & -0.3 \end{pmatrix} and b_0 = (0.1, \; 0.2).

Linear transformation z = x \cdot W_0 + b_0 produces a 4 \times 4 \times 2 tensor. For position x_{0,0} = 1.0:

z_{0,0} = (1.0 \times 0.5 + 0.1, \;\; 1.0 \times (-0.3) + 0.2) = (0.6, \; -0.1)

Applying ReLU: (0.6, \; 0.0). The -0.1 is clamped to zero.

All positions for channel 0 (weight 0.5, bias 0.1) before ReLU:

\begin{pmatrix} 0.6 & 1.1 & 0.35 & -0.4 \\ 1.6 & -0.15 & 0.85 & 1.1 \\ 0.1 & 0.6 & -0.9 & 0.35 \\ 1.1 & 0.35 & 0.6 & -0.15 \end{pmatrix}

After ReLU on channel 0:

\begin{pmatrix} 0.6 & 1.1 & 0.35 & 0.0 \\ 1.6 & 0.0 & 0.85 & 1.1 \\ 0.1 & 0.6 & 0.0 & 0.35 \\ 1.1 & 0.35 & 0.6 & 0.0 \end{pmatrix}

Config entry 2: 'M' (MaxPool 2 \times 2). Each 2 \times 2 block reduces to its maximum. For channel 0:

\begin{pmatrix} \max(0.6, 1.1, 1.6, 0.0) & \max(0.35, 0.0, 0.85, 1.1) \\ \max(0.1, 0.6, 1.1, 0.35) & \max(0.0, 0.35, 0.6, 0.0) \end{pmatrix} = \begin{pmatrix} 1.6 & 1.1 \\ 1.1 & 0.6 \end{pmatrix}

Output shape: (1, 2, 2, 2). Spatial dimensions halved from 4 \times 4 to 2 \times 2, channels changed from 1 to 2. One weight-bias pair was consumed (weight index 0 to 1), and one pooling layer applied (no weights consumed).

VGG Features for Transfer Learning

The VGG feature extractor became one of the most widely reused components in deep learning. Pre-trained VGG-16 and VGG-19 features served as general-purpose image representations for years after publication.

• Neural Style Transfer (Gatys et al., 2015): Used features from multiple VGG-19 layers to define content loss (deep layers) and style loss (Gram matrices of early/mid layers). VGG's uniform architecture produces hierarchical features at predictable spatial scales.
• Perceptual Loss (Johnson et al., 2016): Replaced pixel-wise losses with feature-space distances through a frozen VGG network. This became standard for super-resolution and image synthesis.
• Object Detection (Faster R-CNN, Ren et al., 2015): Used VGG-16 as the backbone, processing the entire image once into a shared feature map for region proposals and classification.
• Semantic Segmentation (FCN, Long et al., 2015): Converted VGG into a fully convolutional network by replacing FC layers with 1 \times 1 convolutions, fusing intermediate features via skip connections.

VGG features transferred well because the extractor builds a clean hierarchy: early layers produce universally useful edge detectors, while deeper layers produce task-adaptable semantic features. Fine-tuning only the last few layers of a pre-trained VGG often matched training from scratch on much smaller datasets.

Pitfalls

• Consuming weights in the wrong order. The weight index must advance only for integer config entries, never for 'M'. If the code increments the weight index for every config entry (including pooling), conv layers receive wrong weight-bias pairs and the final index overshoots the list length. The fix is a separate counter that increments only inside the integer branch.
• Applying pooling when the config says conv (or vice versa). If the dispatch logic checks the type incorrectly, an integer like 64 might be treated as a truthy string and routed to pooling, or 'M' might be compared numerically. The check must distinguish between Python integers and the string 'M' explicitly.
• Wrong input channels for the first conv layer. The first conv weight matrix must have input channels matching the input tensor's channel dimension (typically 3 for RGB). A shape mismatch causes a crash or silent broadcasting errors.
• Forgetting ReLU after convolution. Every integer entry implies Conv followed by ReLU. Omitting ReLU turns the feature extractor into a sequence of linear transformations, which collapses to a single linear map regardless of depth. Without nonlinearity, a 16-layer network has the same representational power as a 1-layer network.
• Spatial dimension mismatch at pooling. Max pooling with 2 \times 2 kernel and stride 2 requires even spatial dimensions. If odd-sized feature maps reach a pooling layer, it either crashes or silently drops the last row/column. VGGNet avoids this by starting from $ \times 224$ with same-padding convolutions.
• Modifying the input tensor in-place. If the feature extractor modifies the input array directly without copying, the caller's data is corrupted. The safe pattern is to copy the input or use operations that return new arrays.

Examples

Hints

1. Keep a separate index for kernels because M consumes no weights.
2. Reuse the same convolution operation for every integer entry.
3. Apply max pooling immediately when M is encountered.

Requirements

• Use NumPy.
• Consume one kernel and bias for each integer entry.
• Apply same-padded convolution and ReLU for integer entries.
• Apply 2 by 2 max pooling for M entries.
• Return a float64 NumPy array.

Constraints

• x uses NHWC layout and dtype float64.
• config contains positive integers and M strings.
• kernels and biases contain one compatible pair per integer entry.
• Spatial dimensions are even whenever pooling occurs.
• Every supplied numeric array has dtype float64.

Starter Code

import numpy as np

def vgg_features(x: np.ndarray, config: list,
                 kernels: list, biases: list) -> np.ndarray:
    """
    Returns the float64 VGG features in NHWC layout.
    """
    pass

Test Cases