Complete VGG Network

Very Deep Convolutional Networks

Easy

Problem

Theory

The complete VGG-16 network is an end-to-end image classification model. It takes a raw image tensor and produces class logits by composing two stages: a feature extractor built from convolutional blocks and a classifier built from fully connected layers. A config list drives the architecture, making the same code reusable across VGG-11 through VGG-19.

This is the capstone of the VGGNet architecture. Every component studied in isolation -- conv blocks, max pooling, feature extraction, and the classifier head -- is assembled here into a single coherent model that maps pixels to predictions.

What It Is / What It Does

VGG-16 is a convolutional neural network for image classification. Given an input image tensor of shape (B, 3, 224, 224), it outputs a logits tensor of shape (B, 1000) representing scores for 1000 ImageNet classes. The pipeline has two stages:

• Stage 1 -- Feature Extraction: The input image passes through 5 convolutional blocks. Each block applies a sequence of 3 \times 3 convolutions with ReLU, followed by 2 \times 2 max pooling. The spatial dimensions halve at each block while channel depth increases: 64 \to 128 \to 256 \to 512 \to 512.
• Stage 2 -- Classification: The feature map is flattened into a vector and passed through 3 fully connected layers. The first two FC layers have 4096 units with ReLU and dropout. The final FC layer projects to 1000 classes.

The output is raw logits -- no softmax is applied. Cross-entropy loss applies log-softmax internally for numerical stability during training.

Key Equations

Stage 1 -- Feature extraction:

\text{features} = \text{vgg\_features}(x, \text{config})

where x \in \mathbb{R}^{B \times 3 \times 224 \times 224} is the input image batch, config is the layer specification list, and \text{features} \in \mathbb{R}^{B \times 512 \times 7 \times 7}.

Stage 2 -- Classification:

\text{logits} = \text{vgg\_classifier}(\text{flatten}(\text{features}))

where \text{flatten}(\text{features}) \in \mathbb{R}^{B \times 25088} and \text{logits} \in \mathbb{R}^{B \times 1000}.

The full forward pass in one line:

\text{logits} = \text{vgg\_classifier}(\text{flatten}(\text{vgg\_features}(x, \text{config})))

The flatten operation reshapes the 4D feature tensor (B, 512, 7, 7) into a 2D matrix (B, 25088) where 25088 = 512 \times 7 \times 7. This bridges the convolutional and fully connected stages.

The Two-Stage Design

VGG cleanly separates feature extraction from classification. This reflects a fundamental design principle that proved enormously influential:

Feature Extraction (convolutional blocks):

• Processes the spatial structure of the image through local receptive fields.
• Translation-equivariant: shifting the input shifts the feature maps by the same amount.
• Parameter-efficient: a 3 \times 3 \times C_{\text{in}} \times C_{\text{out}} conv kernel is reused at every spatial position.

Classification (fully connected layers):

• Takes the fixed-size flattened feature vector and maps it to class scores.
• Not spatially aware -- every input element connects to every output element.
• Dropout between FC layers provides regularization against overfitting.

This clean separation means you can replace the classifier for transfer learning while keeping the feature extractor frozen. You can also discard the classifier entirely and use the feature maps for detection, segmentation, or style transfer.

VGG-16 Architecture

The "16" in VGG-16 counts the number of weight layers: 13 convolutional layers plus 3 fully connected layers. Activation functions, pooling layers, and dropout do not count because they have no learned weights.

The 5 convolutional blocks:

• Block 1: 2 conv layers, 3 \times 3, 64 filters. Output: (B, 64, 112, 112) after pooling.
• Block 2: 2 conv layers, 3 \times 3, 128 filters. Output: (B, 128, 56, 56) after pooling.
• Block 3: 3 conv layers, 3 \times 3, 256 filters. Output: (B, 256, 28, 28) after pooling.
• Block 4: 3 conv layers, 3 \times 3, 512 filters. Output: (B, 512, 14, 14) after pooling.
• Block 5: 3 conv layers, 3 \times 3, 512 filters. Output: (B, 512, 7, 7) after pooling.

The 3 fully connected layers:

• FC1: 25088 \to 4096, ReLU, Dropout(0.5)
• FC2: 4096 \to 4096, ReLU, Dropout(0.5)
• FC3: 4096 \to 1000 (no activation)

The naming convention -- VGG-11, VGG-13, VGG-16, VGG-19 -- always counts weight layers only. All variants share the same classifier; they differ only in how many conv layers appear within each block.

The Config-Driven Approach

A key insight in the VGG paper is that the entire family of networks can be built from a single config list. For VGG-16 (configuration D in the paper), the config is:

\text{config} = [64, 64, \text{M}, 128, 128, \text{M}, 256, 256, 256, \text{M}, 512, 512, 512, \text{M}, 512, 512, 512, \text{M}]

The rules are simple:

• An integer n: Add a Conv2d with $$ output channels, 3 \times 3 kernel, padding 1, followed by ReLU.
• The letter M: Add a 2 \times 2 MaxPool2d with stride 2, halving spatial dimensions.

The same vgg_features function handles every variant:

• VGG-11 (config A): [64, \text{M}, 128, \text{M}, 256, 256, \text{M}, 512, 512, \text{M}, 512, 512, \text{M}] -- 8 conv layers.
• VGG-13 (config B): [64, 64, \text{M}, 128, 128, \text{M}, 256, 256, \text{M}, 512, 512, \text{M}, 512, 512, \text{M}] -- 10 conv layers.
• VGG-16 (config D): 13 conv layers as shown above.
• VGG-19 (config E): Adds a 4th conv layer in blocks 3, 4, and 5 -- 16 conv layers total.

The classifier is identical for all variants because all configs produce the same 512 \times 7 \times 7 spatial output.

Parameter Count

VGG-16 has approximately 138 million parameters. The distribution between the two stages is dramatically uneven:

Convolutional layers (~14.7M parameters):

• Block 1: 3 \times 3 \times 3 \times 64 + 3 \times 3 \times 64 \times 64 \approx 38{,}592
• Block 2: 3 \times 3 \times 64 \times 128 + 3 \times 3 \times 128 \times 128 \approx 221{,}184
• Blocks 3-5 scale up with channel depth, totaling ~14.4M for the remaining layers.

Fully connected layers (~123.6M parameters):

• FC1: 25{,}088 \times 4{,}096 \approx 102.8\text{M}
• FC2: 4{,}096 \times 4{,}096 \approx 16.8\text{M}
• FC3: 4{,}096 \times 1{,}000 \approx 4.1\text{M}

The FC layers contain about 89% of all parameters, with FC1 alone holding 74% of the total. This is why later architectures (GoogLeNet, ResNet) replaced FC layers with global average pooling -- it eliminates the massive FC1 bottleneck entirely. VGG's parameter inefficiency was a key motivation for subsequent architectural innovations.

Paper Context

VGGNet was introduced by Karen Simonyan and Andrew Zisserman of the Visual Geometry Group at Oxford in their 2014 paper "Very Deep Convolutional Networks for Large-Scale Image Recognition." The network was the runner-up in ILSVRC-2014 classification (behind GoogLeNet) but won the localization task.

The key insight: depth matters. The paper systematically evaluated networks from 11 to 19 layers, all using only 3 \times 3 convolutions. This departed from AlexNet (2012), which used larger $ \times 11$ and 5 \times 5 filters. Simonyan and Zisserman showed that stacking two 3 \times 3 convolutions achieves the same receptive field as one 5 \times 5 but with fewer parameters (2 \times 9C^2 = 18C^2 vs. 25C^2) and an extra nonlinearity.

As the paper states: "In spite of a large number of parameters (144 million for VGG-19), these networks require few epochs to converge." The authors attributed this to the implicit regularization provided by greater depth and smaller filter sizes.

Pre-trained VGG weights became a standard tool in computer vision. Before large-scale pre-training with ImageNet-21k or self-supervised methods, VGG-16 pre-trained on ImageNet was the default feature extractor for transfer learning.

Spatial Dimension Trace

Trace an input of shape (1, 3, 224, 224) through the full VGG-16:

Feature extraction stage:

• Input: (1, 3, 224, 224)
• Block 1 convs: 3 \to 64 channels, padding preserves spatial: (1, 64, 224, 224). MaxPool: (1, 64, 112, 112).
• Block 2 convs: 64 \to 128: (1, 128, 112, 112). MaxPool: (1, 128, 56, 56).
• Block 3 convs: 128 \to 256: (1, 256, 56, 56). MaxPool: (1, 256, 28, 28).
• Block 4 convs: 256 \to 512: (1, 512, 28, 28). MaxPool: (1, 512, 14, 14).
• Block 5 convs: 512 \to 512: (1, 512, 14, 14). MaxPool: (1, 512, 7, 7).

Flatten: (1, 512, 7, 7) \to (1, 25088)

Classifier stage:

• FC1 + ReLU + Dropout: (1, 25088) \to (1, 4096)
• FC2 + ReLU + Dropout: (1, 4096) \to (1, 4096)
• FC3: (1, 4096) \to (1, 1000)

The spatial dimensions follow a clean halving pattern: 224 \to 112 \to 56 \to 28 \to 14 \to 7. Each max pool divides by 2. The final $ \times 7$ comes from 224 / 2^5 = 7. This is why VGG requires 224 \times 224 input -- the 5 pooling layers produce a $ \times 7$ grid that yields 512 \times 7 \times 7 = 25{,}088 features matching FC1's expected input.

VGG's Legacy

Despite being superseded by ResNet (2015) for classification accuracy, VGG remains one of the most influential networks in deep learning. Its impact extends well beyond classification:

• Transfer learning backbone: VGG features were the default starting point for object detection (Fast R-CNN, Faster R-CNN) and semantic segmentation (FCN) before ResNet. The two-stage design made it easy to extract features at different spatial resolutions.
• Neural style transfer: Gatys et al. (2015) used VGG feature maps at different layers to capture content (deeper layers) and style (Gram matrices of earlier layers). VGG's uniform 3 \times 3 architecture produces hierarchical features well-suited for this decomposition.
• Perceptual loss: Johnson et al. (2016) replaced pixel-wise loss with VGG feature-space loss for super-resolution and style transfer. This "perceptual loss" compares VGG representations of generated and target images, producing more natural results than pixel MSE.
• Teaching tool: VGG's simplicity -- just 3 \times 3 convolutions, max pooling, and FC layers, with no skip connections, no batch normalization, no inception modules -- makes it ideal for understanding how deep CNNs work.

VGG was superseded because its massive FC layers waste parameters and without skip connections, training beyond 19 layers suffers from vanishing gradients. ResNet solved both problems. But VGG's design philosophy -- simple building blocks, systematic depth scaling, small filters everywhere -- directly informed the architectures that followed.

Pitfalls

1. Using the wrong config for the variant

VGG-16 uses config D: [64, 64, \text{M}, 128, 128, \text{M}, 256, 256, 256, \text{M}, 512, 512, 512, \text{M}, 512, 512, 512, \text{M}]. A common mistake is using config E (VGG-19) which has 4 conv layers in blocks 3-5 instead of 3. If you load VGG-16 pre-trained weights into a VGG-19 architecture, the weight shapes will mismatch and produce wrong results.

2. Forgetting to flatten between features and classifier

The feature extractor outputs a 4D tensor (B, 512, 7, 7) but the classifier expects a 2D tensor (B, 25088). You must explicitly flatten between the two stages. Passing the 4D tensor directly to FC1 causes a shape mismatch error. The flatten operation is $ -1)$ or x.\text{flatten}(1).

3. Wrong weight initialization order

When loading pre-trained weights, the state dict keys must match exactly. A frequent bug is building the feature extractor with layers in a different order than the reference implementation. If your config parsing adds extra layers (like BatchNorm, which original VGG does not use), all subsequent weights will be misaligned. The model runs without errors but produces random predictions.

4. Applying softmax in the forward pass

The model should output raw logits, not probabilities. If you apply softmax in the forward pass and then use nn.CrossEntropyLoss (which applies log-softmax internally), you get double-softmax. The gradients become near-zero and the model fails to learn. Apply softmax outside the model at inference if needed.

5. Incorrect input dimensions

VGG-16 expects 224 \times 224 input. With 5 max-pool layers of stride 2, any input not divisible by $ = 32$ produces non-integer spatial dimensions. Using 256 \times 256 images produces 8 \times 8 \times 512 = 32{,}768 features instead of 25,088, causing a dimension mismatch at FC1.

Examples

Hints

1. Finish all config entries before flattening.
2. Use the next kernel only for an integer config entry.
3. Do not apply ReLU after the final classifier projection.

Requirements

• Use NumPy and every supplied parameter.
• Execute config entries in order.
• Apply ReLU after convolutions and the first two classifier layers.
• Apply pooling only for M entries.
• Return float64 raw logits.

Constraints

• x uses NHWC layout and dtype float64.
• config, kernels, and biases satisfy the feature-extractor contract.
• classifier contains exactly W1, b1, W2, b2, W3, and b3 with compatible float64 shapes.
• Spatial dimensions are even whenever pooling occurs.

Starter Code

import numpy as np

def vgg_forward(x: np.ndarray, config: list, kernels: list,
                biases: list, classifier: dict) -> np.ndarray:
    """
    Returns float64 class logits with shape (B, C_classes).
    """
    pass

Test Cases