Mode Collapse Detection

Generative Adversarial Networks (GAN)

Easy

Problem

Theory

Mode collapse is one of the most common failure modes in GANs. It occurs when the generator produces limited varieties of samples, mapping different noise vectors to nearly identical outputs. Instead of learning the full data distribution, the generator "collapses" onto one or a few modes.

This problem detects mode collapse through the diversity score: the mean per-feature standard deviation of generated samples. When this score falls below a threshold, the generator is flagged as collapsed.

What It Is

Mode collapse detection measures output diversity to identify generator failure. A healthy generator produces varied outputs when given different noise inputs. Sample 100 random noise vectors z_1, \ldots, z_{100} and the resulting G(z_1), \ldots, G(z_{100}) should span the data distribution. In a collapsed generator, these outputs cluster tightly around one or a few points regardless of the input noise.

The detection mechanism computes a diversity score from a generated batch. Each feature dimension is examined independently for variation across the batch. If the generator has collapsed, every feature has near-zero variance. The diversity score aggregates per-feature variation into a single number compared against a threshold to produce a boolean is_collapsed signal.

This is a diagnostic tool, not a training objective. It tells you whether the generator has failed but does not fix the problem. Think of it as a smoke detector for GANs.

Key Equations

Given N generated samples, each with D features, arranged as X \in \mathbb{R}^{N \times D}:

Per-feature mean:

\mu_j = \frac{1}{N} \sum_{i=1}^{N} X_{i,j}

Per-feature standard deviation:

\sigma_j = \sqrt{\frac{1}{N} \sum_{i=1}^{N} (X_{i,j} - \mu_j)^2}

This is the population standard deviation (dividing by N, not N-1).

Diversity score (mean of per-feature stds):

\text{diversity\_score} = \frac{1}{D} \sum_{j=1}^{D} \sigma_j

Collapse detection:

\text{is\_collapsed} = \begin{cases} \text{True} & \text{if } \text{diversity\_score} < \text{threshold} \\ \text{False} & \text{otherwise} \end{cases}

The threshold depends on the data domain. For normalized data (features in [0, 1]), a threshold around 0.1 is common. For standardized data, thresholds around 0.5 may be appropriate.

What Mode Collapse Looks Like

All outputs nearly identical despite different noise inputs. The generator learns to ignore z and maps everything to roughly the same output: G(z_1) \approx G(z_2) \approx \cdots \approx G(z_N) for all noise vectors. The generator becomes essentially a constant function. If generating images, every image looks the same. If generating tabular data, every row has nearly identical values.

The generator ignores z. A well-functioning generator uses noise as a source of randomness, mapping different latent regions to different data regions. A collapsed generator develops weights that cancel out z's influence. The hidden layers produce similar activations regardless of input, and the information bottleneck discards diversity rather than carrying it through.

Everything maps to one or a few modes. Real distributions are multimodal. Consider handwritten digits with 10 modes (digits 0-9). A fully collapsed generator might produce only "1" because that single mode temporarily fools the discriminator. A partially collapsed generator might produce only "1" and "7". In both cases the generator fails to capture the full distribution.

Why Mode Collapse Happens

Mode collapse arises from the adversarial training dynamics between generator and discriminator:

• Generator finds one output that fools D. If the generator discovers a single point that consistently receives high "real" scores, it is incentivized to map all noise vectors there. From G's perspective, this achieves low loss without modeling the full distribution.
• Exploitation over exploration. The generator can explore the data space to cover all modes (risky, slow) or exploit a known good mode (safe, fast). Gradient descent favors exploitation because gradients point toward the nearest "real-looking" region, not toward uncovered modes.
• Discriminator cannot enforce mode coverage. D evaluates individual samples, not the distribution. It can judge whether one sample looks real but has no mechanism to say "you are repeating yourself." Per-sample feedback cannot express distributional constraints.
• Cycling rather than converging. The minimax game can cycle: G collapses to mode A, D rejects A, G switches to mode B, D rejects B, G switches back. The generator covers one mode at a time, rotating rather than covering all modes simultaneously.

Types of Mode Collapse

• Complete collapse (single output). The generator maps every noise vector to a single point: G(z) = c for all z. Diversity score is near zero. Easy to detect but rare in practice because even a weak discriminator can reject a single repeated sample. Usually indicates a training bug.
• Partial collapse (few clusters). The generator produces samples from k modes where k is much smaller than the true count. For MNIST, it might only produce digits 1, 4, and 7. Diversity score is moderate: higher than complete collapse due to inter-cluster variation but lower than a healthy generator. This is the most common form.
• Intra-class collapse (limited variation within each mode). The generator covers all major modes but with minimal variation within each. For MNIST, all 10 digits appear but each looks nearly identical across samples: same stroke width, slant, position. The diversity score may look acceptable because inter-mode variation masks missing intra-mode variation. Hardest form to detect with simple metrics.

Paper Context

Goodfellow et al. (2014) introduced GANs with the minimax formulation:

\min_G \max_D \; \mathbb{E}_{x \sim p_{\text{data}}}[\log D(x)] + \mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))]

The paper proves this objective has a global optimum where p_G = p_{\text{data}}. However, the proof assumes infinite capacity and perfect optimization. In practice, a finite neural network may lack capacity to represent p_{\text{data}}, and gradient-based optimization may not find the global optimum. Mode collapse is a direct consequence of this gap between theory and practice.

The paper acknowledges that training requires careful balancing of G and D updates. When D is too strong, G receives vanishing gradients. When D is too weak, G receives noisy gradients. Both scenarios can trigger or worsen mode collapse.

Subsequent work addressing mode collapse includes minibatch discrimination (Salimans et al., 2016), unrolled GANs (Metz et al., 2017), Wasserstein GANs (Arjovsky et al., 2017), and spectral normalization (Miyato et al., 2018). Each attacks the problem from a different angle but none eliminates it entirely.

The Standard Deviation Metric

Standard deviation is a proxy for diversity because it directly measures spread. High std along a feature means that feature takes a wide range of values across the batch, indicating diversity. Near-zero std means all samples share nearly the same value, indicating collapse.

Why per feature then average? Different features may have different natural scales. A single global std would be dominated by high-magnitude features. Computing per-feature stds and averaging gives each feature equal weight in the diversity score.

What low std means geometrically. Each generated sample is a point in \mathbb{R}^D. A healthy generator scatters points throughout the region occupied by real data. Low std along all features means all points cluster in a tiny hyperrectangle. In the extreme (\sigma_j = 0 for all j), all points occupy a single location. Mode collapse shrinks the point cloud from a region to a point or a few isolated clusters.

Averaging vs. minimum. The mean of per-feature stds provides a smoother signal than \min_j \sigma_j. The minimum is more conservative (flags collapse if any single feature collapses) but more sensitive to noise. The mean is less prone to false positives from individual noisy features.

Numerical Example

Generator with D = 3 features, batch of N = 5 samples.

Collapsed generator output:

Per-feature means: \mu_1 = 0.508, \mu_2 = 0.820, \mu_3 = 0.340.

Feature 1 deviations: [0.002, -0.008, 0.012, -0.008, 0.002]. Squared: [0.000004, 0.000064, 0.000144, 0.000064, 0.000004]. Mean: 0.000056. \sigma_1 = \sqrt{0.000056} \approx 0.0075.

Feature 2: deviations [0.00, -0.01, 0.01, 0.00, 0.00]. Mean of squares: 0.00004. \sigma_2 \approx 0.0063.

Feature 3: deviations [0.00, 0.01, -0.01, 0.00, 0.00]. Mean of squares: 0.00004. \sigma_3 \approx 0.0063.

Diversity score: (0.0075 + 0.0063 + 0.0063) / 3 \approx 0.0067.

With threshold = 0.1: 0.0067 < 0.1 \Rightarrow \text{is\_collapsed} = \text{True}. All five samples are nearly identical.

Healthy generator output:

Per-feature stds: \sigma_1 \approx 0.299, \sigma_2 \approx 0.253, \sigma_3 \approx 0.196.

Diversity score: (0.299 + 0.253 + 0.196) / 3 \approx 0.249. With threshold = 0.1: 0.249 > 0.1 \Rightarrow \text{is\_collapsed} = \text{False}. The generator produces varied samples.

Mitigation Strategies

Once mode collapse is detected, several techniques can address it:

• Minibatch discrimination (Salimans et al., 2016). The discriminator receives pairwise similarity information across the minibatch. If G produces identical outputs, D sees high similarity and penalizes it. This addresses the limitation that the standard D evaluates samples independently.
• Unrolled GANs (Metz et al., 2017). G's loss is computed by unrolling several D update steps into the future. G optimizes against a future discriminator that has adapted to its outputs, discouraging exploitation of temporary weaknesses at a single mode.
• Wasserstein distance (Arjovsky et al., 2017). Replaces the JS divergence with the Earth Mover's distance, providing meaningful gradients even when generator and real distributions have non-overlapping support. The smoother loss landscape reduces mode collapse.
• Spectral normalization (Miyato et al., 2018). Constrains the Lipschitz constant of D by normalizing spectral norms of weight matrices. Prevents D from becoming too sharp, stabilizing training and reducing cycling behavior.
• Diversity-promoting losses. Add explicit diversity terms to G's loss, e.g., a penalty proportional to -\text{diversity\_score}. Mode regularization penalizes small pairwise distances between generated samples, treating diversity as a first-class objective.

Pitfalls

1. Threshold too high gives false positives.

If real data naturally has low diversity in some features (binary features, categorical features with few values), the diversity score of a perfect generator would also be low. The threshold must be calibrated relative to the real data's diversity, not set arbitrarily.

2. Threshold too low misses partial collapse.

A threshold of 0.01 catches complete collapse but misses a generator stuck on a handful of modes. Inter-mode variation can inflate per-feature stds, making a partially collapsed generator look healthy.

3. Std of 0 does not always mean collapse.

If the real data has a constant feature (e.g., a bias column always equal to 1.0), then \sigma_j = 0 for that feature is correct behavior. A perfect generator should reproduce constant features exactly. Interpreting zero std as collapse without checking the real distribution leads to false alarms.

4. Confusing low quality with mode collapse.

A generator can produce diverse but unrealistic samples (high diversity, low quality) or one very realistic sample repeatedly (low diversity, high quality). The diversity score measures only spread, not realism. A high score does not mean the GAN works well.

5. Batch size too small for reliable std.

With N = 5 samples, the std estimate has high variance. A generator might appear collapsed because a small batch happened to produce similar outputs by chance. Use batches of at least 100-500 samples for reliable detection. The metric is meant for substantial batches, not training minibatches of size 8 or 16.

Examples

Hints

1. Reduce the standard deviation over axis zero.
2. Average the resulting per-feature values before comparing with the threshold.

Requirements

• Use NumPy.
• Compute population standard deviations across the batch axis.
• Average the per-feature deviations.
• Use a strict comparison with the supplied threshold.
• Return exactly the documented dictionary.

Constraints

• generated_samples has shape (N, D).
• 2 \le N \le 9 and 1 \le D \le 6.
• threshold is a positive finite float.
• All sample values are finite real numbers.

Starter Code

import numpy as np

def detect_mode_collapse(generated_samples: np.ndarray, threshold: float = 0.1) -> dict:
    """
    Returns diversity_score and is_collapsed in a dictionary.
    """
    pass

Test Cases