GAN Discriminator

Generative Adversarial Networks (GAN)

Medium

Problem

Theory

The discriminator is the binary classifier at the heart of a Generative Adversarial Network. Given an input x, it outputs a scalar probability D(x) \in [0, 1] estimating whether x came from the real training data or was fabricated by the generator. Values near 1 mean "probably real," values near 0 mean "probably fake." This probability drives the adversarial training dynamic introduced in Goodfellow et al. (2014).

What It Is: A Binary Classifier

The discriminator distinguishes between two classes: real samples drawn from p_{\text{data}}(x), and fake samples produced by the generator G(z) where z \sim p_z(z) is a latent noise vector.

In its simplest form, the discriminator is a single-layer feedforward network. For input x \in \mathbb{R}^d:

D(x) = \sigma(x \cdot W)

where W \in \mathbb{R}^{d \times 1} is a learnable weight matrix and \sigma is the sigmoid function. The weight matrix projects the d-dimensional input to a single scalar, and sigmoid squashes it into a probability.

During training, the discriminator receives inputs from two sources:

• Real samples x \sim p_{\text{data}}(x): drawn from the training dataset. Target: D(x) \approx 1.
• Fake samples \tilde{x} = G(z): produced by the generator. Target: D(G(z)) \approx 0.

The discriminator does not know which source a given sample comes from. It must learn to distinguish real from fake purely from the statistical properties of the data.

Key Equations

The Sigmoid Function

The sigmoid maps any real-valued input to (0, 1):

\sigma(a) = \frac{1}{1 + e^{-a}}

Properties relevant to the discriminator:

• \sigma(0) = 0.5: The decision boundary. Maximum uncertainty.
• \sigma(a) \to 1 as a \to +\infty: High confidence the input is real.
• \sigma(a) \to 0 as a \to -\infty: High confidence the input is fake.
• Derivative: \sigma'(a) = \sigma(a)(1 - \sigma(a)), maximized at a = 0, vanishing at extremes.

Discriminator Forward Pass

For input x \in \mathbb{R}^d and weight W \in \mathbb{R}^{d \times 1}:

Step 1. Compute the logit: a = x \cdot W, producing a scalar a \in \mathbb{R}.

Step 2. Apply sigmoid: D(x) = \sigma(a) = \frac{1}{1 + e^{-a}}.

The logit a is a log-odds ratio: a = \log \frac{D(x)}{1 - D(x)}.

Output Interpretation

• D(x) = 0.95: 95% confident $$ is real. Logit a \approx 2.94.
• D(x) = 0.50: Maximum uncertainty. Logit a = 0.
• D(x) = 0.05: 95% confident $$ is fake. Logit a \approx -2.94.

The Discriminator's Role in the Minimax Game

The GAN objective from Goodfellow et al. (2014) is a two-player minimax game:

\min_G \max_D \; V(D, G) = \mathbb{E}_{x \sim p_{\text{data}}(x)}[\log D(x)] + \mathbb{E}_{z \sim p_z(z)}[\log(1 - D(G(z)))]

The discriminator maximizes V(D, G). The two terms from the discriminator's perspective:

• \mathbb{E}_{x \sim p_{\text{data}}}[\log D(x)]: For real samples, the discriminator wants D(x) \to 1, making \log D(x) \to 0 (its maximum). Small D(x) yields a large negative penalty.
• \mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))]: For fake samples, the discriminator wants D(G(z)) \to 0, making \log(1 - D(G(z))) \to 0. Being fooled (D(G(z)) large) yields a large negative penalty.

In practice, this is equivalent to minimizing binary cross-entropy with label 1 for real and 0 for fake:

\mathcal{L}_D = -\frac{1}{m} \sum_{i=1}^{m} \left[ \log D(x^{(i)}) + \log(1 - D(G(z^{(i)}))) \right]

The discriminator performs gradient descent on \mathcal{L}_D, updating weights to better separate real from fake. The generator, conversely, minimizes V by making D(G(z)) large (fooling the discriminator). This adversarial tension drives GAN learning.

Why Sigmoid

Probability Output

Sigmoid maps the raw logit to a valid probability in (0, 1). This is essential because the value function involves \log D(x) and \log(1 - D(G(z))), both requiring D(x) \in (0, 1) to avoid logarithms of non-positive numbers.

Connection to Binary Cross-Entropy

The canonical pairing in logistic regression is sigmoid + BCE loss. When combined, the gradient with respect to the logit a simplifies to:

\frac{\partial \mathcal{L}}{\partial a} = D(x) - y

where y \in \{0, 1\} is the label. The gradient is simply prediction minus target, with no \sigma'(a) factor that could cause vanishing gradients in logit space. This clean gradient is why sigmoid + BCE is the standard for binary classification.

Log-Likelihood Interpretation

The value function V(D, G) is the expected log-likelihood of the discriminator under a Bernoulli model: P(\text{real} | x) = D(x) and P(\text{fake} | x) = 1 - D(x). Sigmoid ensures this probabilistic interpretation is valid.

Paper Context: Goodfellow et al. (2014)

In "Generative Adversarial Nets," the discriminator is described: "The discriminative model D estimates the probability that a sample came from the training data rather than G." The paper frames GANs through an analogy: the generator is a counterfeiter producing fake currency, the discriminator is police detecting counterfeits, and both improve through competition.

The paper uses multilayer perceptrons for both G and D. The theoretical analysis assumes D has enough capacity to represent the optimal discriminator for any G. Goodfellow et al. prove that for fixed G, the optimal discriminator is:

D^*_G(x) = \frac{p_{\text{data}}(x)}{p_{\text{data}}(x) + p_g(x)}

where p_g(x) is the generator's induced distribution. The optimal discriminator compares data density to generator density: when p_{\text{data}}(x) > p_g(x), D^*(x) > 0.5; when p_g(x) > p_{\text{data}}(x), D^*(x) < 0.5. This result underpins the paper's main theorem: the global minimum of V(D^*, G) is achieved if and only if p_g = p_{\text{data}}, at which point V = -\log 4.

The Optimal Discriminator

Derivation Sketch

For fixed G, rewrite V as an integral:

V(D, G) = \int_x \left[ p_{\text{data}}(x) \log D(x) + p_g(x) \log(1 - D(x)) \right] dx

At each point x, let a = p_{\text{data}}(x) and b = p_g(x). We maximize f(D) = a \log D + b \log(1 - D):

f'(D) = \frac{a}{D} - \frac{b}{1 - D} = 0 \implies D^* = \frac{a}{a + b} = \frac{p_{\text{data}}(x)}{p_{\text{data}}(x) + p_g(x)}

The second derivative f''(D) = -a/D^2 - b/(1-D)^2 < 0 for positive a, b, confirming a maximum.

At Equilibrium: D^* = 0.5 Everywhere

When the generator perfectly matches the data distribution (p_g = p_{\text{data}}):

D^*(x) = \frac{p_{\text{data}}(x)}{p_{\text{data}}(x) + p_{\text{data}}(x)} = \frac{1}{2}

At Nash equilibrium, the discriminator outputs 0.5 for every input. It cannot distinguish real from fake because the distributions are identical. This is counterintuitive: perfect GAN training makes the discriminator useless. Its purpose is not to be a good classifier at convergence, but to provide a useful training signal to the generator along the way.

Numerical Example

Consider a single-layer discriminator with d = 3 and weight vector W = [0.8, -1.2, 0.5]^T.

Real Sample

Let x_{\text{real}} = [2.0, -1.0, 1.5] from the training data.

Step 1. Logit: a = (2.0)(0.8) + (-1.0)(-1.2) + (1.5)(0.5) = 1.6 + 1.2 + 0.75 = 3.55

Step 2. Sigmoid: D(x_{\text{real}}) = \frac{1}{1 + e^{-3.55}} = \frac{1}{1.0287} \approx 0.972

Output 0.972: very high confidence this is real. Correct.

Fake Sample

Let x_{\text{fake}} = G(z) = [-0.5, 1.8, -0.3] from the generator.

Step 1. Logit: a = (-0.5)(0.8) + (1.8)(-1.2) + (-0.3)(0.5) = -0.4 - 2.16 - 0.15 = -2.71

Step 2. Sigmoid: D(x_{\text{fake}}) = \frac{1}{1 + e^{2.71}} = \frac{1}{16.03} \approx 0.062

Output 0.062: high confidence this is fake. Correct.

Loss Computation

For this mini-batch of one real and one fake sample:

\mathcal{L}_D = -\frac{1}{2}\left[\log(0.972) + \log(1 - 0.062)\right] = -\frac{1}{2}\left[-0.0284 + (-0.0640)\right] = 0.0462

This small loss confirms good performance. A perfect discriminator achieves loss 0. A random discriminator (outputting 0.5 for everything) gets \log 2 \approx 0.693.

Discriminator Evolution

The discriminator architecture has evolved significantly since the original MLP design.

Linear and MLP Discriminators (2014)

The original paper used MLPs with maxout activations. These worked on simple datasets (MNIST) but struggled with complex images because fully connected layers cannot exploit spatial structure and require prohibitively many parameters for large inputs.

CNN Discriminators: DCGAN (2015)

Radford et al. (2015) replaced fully connected layers with strided convolutions. The discriminator uses conv layers with increasing channels, batch normalization, and LeakyReLU, followed by sigmoid. This leverages spatial locality and parameter sharing, enabling realistic 64x64 image generation.

PatchGAN (2016)

Isola et al. (2016) introduced a discriminator that outputs a grid of probabilities instead of a single scalar. Each element classifies whether a local patch is real or fake. This focuses on high-frequency structure (textures, edges) rather than global composition, producing sharper outputs.

Spectral Normalization (2018)

Miyato et al. (2018) constrained the Lipschitz constant by dividing each weight matrix by its largest singular value. This prevents the discriminator from changing too rapidly, ensuring smoother gradients for the generator and stabilizing training.

The Critic in WGAN (2017)

Arjovsky et al. (2017) replaced the discriminator with a "critic" outputting an unbounded real number (no sigmoid). The critic estimates Wasserstein distance between distributions. Lipschitz continuity is enforced through weight clipping or gradient penalty (WGAN-GP, Gulrajani et al., 2017). This addressed training instability and mode collapse.

Pitfalls

Discriminator Too Strong: Gradient Vanishing

If the discriminator becomes too powerful, it outputs values near 1.0 for all real and near 0.0 for all fake. When D(G(z)) \approx 0, the generator's gradient through \log(1 - D(G(z))) vanishes because \log(1 - D(G(z))) \approx \log(1) = 0 is already near its maximum. The generator receives no learning signal. This is the most common GAN failure mode.

Mitigations: train the generator more steps per discriminator step, use label smoothing (targets of 0.9 instead of 1.0 for real), or switch to the non-saturating loss -\log D(G(z)) which provides stronger gradients early in training.

Discriminator Too Weak: No Signal

If the discriminator is underpowered (too few parameters, unstable learning rate, insufficient training), it provides random gradients. The generator has no meaningful signal to improve against, producing noisy, incoherent outputs. The discriminator must be strong enough to provide a meaningful landscape but not so strong that the landscape becomes flat.

Wrong Sigmoid Application

A common bug is applying sigmoid twice: once in the forward pass and again in the loss. Many frameworks provide BCEWithLogitsLoss which internally applies sigmoid. If the network already has sigmoid and you use BCEWithLogitsLoss, the double sigmoid compresses output near 0.5, severely degrading training. Either use BCELoss with sigmoid in the network, or BCEWithLogitsLoss with raw logits. Never both.

Forgetting That D Provides G's Training Signal

The generator never sees real data directly. Its only information about p_{\text{data}} comes from gradients through the discriminator. If the discriminator has blind spots (regions where it cannot distinguish real from fake), the generator has no incentive to improve there. The discriminator's quality directly bounds the generator's potential.

Examples

Hints

1. Use matrix multiplication to obtain one logit per row.
2. Apply the sigmoid elementwise with NumPy broadcasting.

Requirements

• Use NumPy.
• Apply the supplied weight matrix to every sample.
• Apply the sigmoid function elementwise to the logits.
• Return a float64 NumPy array with shape (B, 1).

Constraints

• X has shape (B, D).
• W has shape (D, 1).
• 1 \le B \le 5 and 1 \le D \le 6.
• All input values are finite real numbers.

Starter Code

import numpy as np

def discriminator(x: np.ndarray, W: np.ndarray) -> np.ndarray:
    """
    Returns discriminator probabilities as a float64 array with shape (B, 1).
    """
    pass

Test Cases