GAN Generator

Generative Adversarial Networks (GAN)

Easy

Problem

Theory

The generator is the creative half of a Generative Adversarial Network. It takes a random noise vector and transforms it into a synthetic data sample that is indistinguishable from real data. In Goodfellow et al. (2014), the generator is a differentiable function G(z; \theta_g) trained to fool a companion discriminator.

What It Is

A GAN generator maps a low-dimensional noise space to a high-dimensional data space. You feed it a random vector z, and it outputs a synthetic sample G(z) in the same space as your training data. The generator never sees real data directly; it receives its training signal entirely through the discriminator's gradients.

In this problem, the generator is a single linear layer followed by tanh:

• Input: A noise vector z of shape (noise\_dim,) sampled from a standard Gaussian \mathcal{N}(0, I).
• Transformation: A linear projection z \cdot W + b where W has shape (noise\_dim, output\_dim) and b is zero-initialized with shape (output\_dim,).
• Output: The tanh activation applied element-wise, producing values bounded in [-1, 1].

This architecture is deliberately minimal. It isolates the core concept: mapping noise to data through a learned linear transformation with a bounded nonlinearity.

Key Equations

The generator function for this problem is defined as:

G(z) = \tanh(z \cdot W + b)

where the components are:

• Noise sampling: z \sim \mathcal{N}(0, I) means each component is drawn independently from a standard normal (mean 0, variance 1).
• Linear projection: z \cdot W + b multiplies z of shape (1, noise\_dim) with W of shape (noise\_dim, output\_dim) and adds bias b.
• Tanh activation: Applied element-wise to each component of the linear output.

The tanh function itself is defined as:

\tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}

This can also be written in terms of the sigmoid function \sigma(x) = \frac{1}{1+e^{-x}}:

\tanh(x) = 2\sigma(2x) - 1

Key properties of tanh relevant to the generator:

• Range: Output is strictly bounded in (-1, 1), approaching \pm 1 asymptotically.
• Zero-centered: \tanh(0) = 0, and the function is symmetric around the origin.
• Derivative: \frac{d}{dx}\tanh(x) = 1 - \tanh^2(x). Maximized at x=0 (equals 1), approaches 0 as |x| grows.
• Saturation: For |x| > 3, the output is nearly \pm 1 and the gradient is nearly zero.

The Noise Space

The noise vector z is the generator's only source of randomness. Each unique z produces a unique output G(z), so the noise space defines the full repertoire of samples the generator can produce.

Why Gaussian Noise

The standard Gaussian \mathcal{N}(0, I) is chosen for several practical reasons:

• Unbounded support: Gaussian noise can take any real value, giving the generator access to the full input range of the linear layer.
• Rotational symmetry: The isotropic Gaussian has equal density in every direction from the origin. No direction in noise space is privileged before training.
• Concentration of measure: In high dimensions, Gaussian samples concentrate on a thin shell at radius \approx \sqrt{d}, providing well-structured inputs.
• Reparameterization: Gaussian noise is trivially reparameterizable via z = \mu + L\epsilon where \epsilon \sim \mathcal{N}(0, I), keeping gradients flowing.

What Different z Vectors Produce

Before training, different z vectors produce random, meaningless outputs because W is randomly initialized. After training, the generator organizes the noise space so that:

• Nearby z vectors produce similar outputs. G is continuous (both the linear layer and tanh are continuous), so small changes in z yield small changes in G(z).
• Linear interpolation produces smooth transitions. Walking along z_t = (1-t)z_1 + tz_2 generates a smooth morphing sequence between G(z_1) and G(z_2).
• Directions encode meaningful variations. In well-trained generators, specific directions correspond to semantic attributes like brightness or shape.

The noise dimension d acts as a bottleneck. Too small and the generator cannot represent enough variation. Too large and many dimensions become redundant. Common choices range from 64 to 512.

Why Tanh

The choice of tanh as the generator's output activation is deliberate and connects to how data is preprocessed for GAN training.

Bounding Output to [-1, 1]

Without an output activation, z \cdot W + b can produce any real value. Real data occupies a bounded range, so tanh forces every output into (-1, 1), matching the preprocessed data.

Pixel Normalization Convention

Raw pixel values in [0, 255] are normalized to [-1, 1] via x_{norm} = \frac{x}{127.5} - 1. Tanh's output range matches this convention. To convert back: x_{pixel} = (G(z) + 1) \times 127.5.

Comparison With No Activation

Removing tanh entirely makes the generator output unbounded, creating three problems:

• Distribution mismatch: Real data in [-1, 1] versus unbounded generator output gives the discriminator a trivial signal. It can reject any sample outside [-1, 1] without learning the data distribution.
• Training instability: Unbounded outputs can grow arbitrarily large, causing overflow and exploding gradients.
• No implicit regularization: Tanh's saturation provides a soft constraint preventing output drift.

Sigmoid maps to (0, 1) and works with [0, 1]-normalized data, but tanh is preferred because its zero-centered output produces better-conditioned gradients for the discriminator.

Paper Context

The generator concept originates from "Generative Adversarial Nets" by Goodfellow et al., published at NeurIPS 2014.

The Minimax Game

GANs frame generative modeling as a two-player game. The generator G tries to produce realistic samples. The discriminator D tries to distinguish real from generated. The objective is:

\min_G \max_D \; \mathbb{E}_{x \sim p_{data}}[\log D(x)] + \mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))]

From the generator's perspective:

• D(G(z)) is the discriminator's probability estimate that G(z) is real. The generator wants this close to 1.
• \log(1 - D(G(z))) is the generator's loss. When D(G(z)) \to 1 (fooled), this goes to -\infty. When D(G(z)) \to 0 (caught), it goes to 0.
• The generator minimizes this expression by making D(G(z)) as large as possible.

Training Signal From the Discriminator

The generator never sees real data directly. Its learning signal comes from backpropagating through D: G produces G(z), feeds it to D, and D's loss gradient flows backward through D into G. This indirect learning is unique to GANs, unlike autoencoders (reconstruction loss) or VAEs (ELBO maximization).

Practical Training Modification

Goodfellow et al. noted that \log(1 - D(G(z))) provides weak gradients early in training when D(G(z)) \approx 0. They proposed maximizing \log(D(G(z))) instead, where \frac{d}{dx}\log(x) at x \approx 0 is large, giving a stronger learning signal without changing the equilibrium.

Numerical Example

Let us trace a concrete forward pass with noise\_dim = 3 and output\_dim = 4.

Setup

Noise vector (sampled from \mathcal{N}(0, I)):

z = [0.5, -1.2, 0.8]

Weight matrix W of shape (3, 4):

W = \begin{bmatrix} 0.3 & -0.5 & 0.7 & 0.1 \\ -0.4 & 0.6 & -0.2 & 0.8 \\ 0.1 & -0.3 & 0.5 & -0.6 \end{bmatrix}

Bias b = [0, 0, 0, 0] (zero-initialized).

Step 1: Linear Projection

Compute z \cdot W by taking dot products of z with each column of W:

Column 0: (0.5)(0.3) + (-1.2)(-0.4) + (0.8)(0.1) = 0.15 + 0.48 + 0.08 = 0.71

Column 1: (0.5)(-0.5) + (-1.2)(0.6) + (0.8)(-0.3) = -0.25 - 0.72 - 0.24 = -1.21

Column 2: (0.5)(0.7) + (-1.2)(-0.2) + (0.8)(0.5) = 0.35 + 0.24 + 0.40 = 0.99

Column 3: (0.5)(0.1) + (-1.2)(0.8) + (0.8)(-0.6) = 0.05 - 0.96 - 0.48 = -1.39

z \cdot W + b = [0.71, -1.21, 0.99, -1.39]

Step 2: Apply Tanh Element-wise

Element 0: \tanh(0.71) = \frac{e^{0.71} - e^{-0.71}}{e^{0.71} + e^{-0.71}} = \frac{2.034 - 0.492}{2.034 + 0.492} = \frac{1.542}{2.526} = 0.6104

Element 1: \tanh(-1.21) = -\tanh(1.21) = -\frac{3.353 - 0.298}{3.353 + 0.298} = -\frac{3.055}{3.651} = -0.8368

Element 2: \tanh(0.99) = \frac{2.691 - 0.372}{2.691 + 0.372} = \frac{2.319}{3.063} = 0.7571

Element 3: \tanh(-1.39) = -\tanh(1.39) = -\frac{4.015 - 0.249}{4.015 + 0.249} = -\frac{3.766}{4.264} = -0.8832

Final Output

G(z) = [0.6104, -0.8368, 0.7571, -0.8832]

Every value is within (-1, 1). Notice how pre-activations of moderate magnitude (\pm 0.71 to \pm 1.39) already produce outputs fairly close to the tanh boundaries. This illustrates tanh's aggressive range compression beyond \pm 1.

Generator Architecture Evolution

The single-layer generator in this problem is the simplest possible architecture. The field has evolved through several major innovations.

Linear Generators (2014)

The original GAN paper used multilayer perceptrons with ReLU hidden layers and tanh output. These fully connected generators could produce simple samples (MNIST digits) but struggled with structured data because they lack spatial inductive bias. Every output pixel depends on every noise component through dense connections.

DCGAN and Transposed Convolutions (2015)

Radford, Metz, and Chintala introduced DCGANs, replacing dense layers with transposed convolutions. Key guidelines:

• Transposed convolutions for upsampling, starting from a small spatial feature map and progressively upsampling to target resolution.
• Batch normalization in both networks to stabilize training.
• ReLU in hidden layers, tanh in output. This became the standard convention.

DCGAN enabled 64x64 image generation and showed that the latent space supports arithmetic (e.g., "man with glasses" minus "man" plus "woman" yields "woman with glasses").

StyleGAN (2018-2020)

Karras et al. introduced a mapping network transforming z into intermediate latent space w, then injecting w via adaptive instance normalization at each resolution level. StyleGAN2 replaced AdaIN with weight demodulation for improved quality. These architectures achieved photorealistic face generation at 1024x1024.

Diffusion Models Replacing GANs (2020-present)

Denoising diffusion models have largely overtaken GANs for image generation. Instead of a direct noise-to-data mapping, they learn iterative denoising over many steps. They avoid adversarial training instabilities, produce more diverse samples, and achieve better distribution coverage. The tradeoff is inference speed: diffusion requires many denoising steps whereas a GAN generator produces a sample in one forward pass.

Pitfalls

Mode Collapse

The most common GAN failure. The generator maps all noise vectors to a small set of outputs that fool D, ignoring the rest of the data distribution. The minimax objective does not explicitly reward diversity, so if a few prototypical samples consistently fool D, there is no pressure to explore further.

Signs: nearly identical generated samples, low variance across z inputs, oscillating D loss. Mitigations include minibatch discrimination, unrolled optimization, and Wasserstein GAN formulations.

Vanishing Gradients When D Is Too Strong

If D becomes too powerful, it classifies all fakes with high confidence: D(G(z)) \approx 0. Then \log(1 - D(G(z))) \approx 0 provides nearly zero gradient, and G stops learning. The discriminator-to-generator step ratio matters: too many D steps per G step creates this imbalance. The \log(D(G(z))) alternative helps but can cause instability from unbounded gradient magnitude.

Wrong Weight Initialization

If W is too large, pre-activations push tanh into saturation where gradients vanish. If too small, all outputs cluster near zero (\tanh(x) \approx x for small x) and the generator ignores z. Xavier/Glorot initialization sets W_{ij} \sim \mathcal{N}(0, \frac{2}{fan\_in + fan\_out}), keeping pre-activation variance stable.

Forgetting That Tanh Bounds Output

If real data is in [0, 255] or [0, 1] but G outputs (-1, 1), D trivially distinguishes real from fake by checking value ranges alone. Always match normalization to the output activation: tanh with [-1, 1], sigmoid with [0, 1]. Similarly, invert normalization when displaying generated samples, or images will look washed out even when G is performing well.

Examples

Hints

1. Broadcast the bias across the rows of the matrix product.
2. NumPy applies tanh elementwise to the projected batch.

Requirements

• Use NumPy.
• Apply the supplied affine transformation to every noise vector.
• Apply the hyperbolic tangent elementwise.
• Return a float64 NumPy array with shape (B, D).

Constraints

• Z has shape (B, N).
• W has shape (N, D).
• b has shape (D).
• 1 \le B \le 4, 1 \le N \le 5, and 1 \le D \le 6.
• All input values are finite real numbers.

Starter Code

import numpy as np

def generator(z: np.ndarray, W: np.ndarray, b: np.ndarray) -> np.ndarray:
    """
    Returns generated samples as a float64 array with shape (B, D).
    """
    pass

Test Cases