AI Engineer Exhaustive
MediumLoss Functions

Implement Contrastive Loss (Siamese)

Loss Functions

Medium

Problem

Compute contrastive loss for pairs of embeddings. A label of 1 marks a similar pair, while 0 marks a dissimilar pair. First compute each Euclidean distance:

d_i = \lVert a_i-b_i \rVert_2

Then compute each pair loss:

\ell_i = y_id_i^2 + (1-y_i)\max(0,m-d_i)^2

Here, a_i and b_i are the two embeddings, d_i is their distance, y_i is the pair label, m is the margin, and \ell_i is the pair loss. A one-dimensional embedding represents one pair; a two-dimensional input contains one pair per row. Return the mean or sum as a Python float according to reduction.

Theory

Contrastive learning trains a model to produce embeddings where:

This is different from classification:

The Contrastive Loss Formula

For a pair of samples with label y:

L = (1 - y) \cdot \frac{1}{2} D^2 + y \cdot \frac{1}{2} \max(0, m - D)^2

Where:

Breaking Down the Two Cases

Case 1: Similar pair (y = 0)

L = \frac{1}{2} D^2

Case 2: Dissimilar pair (y = 1)

L = \frac{1}{2} \max(0, m - D)^2

The Role of the Margin

The margin m defines the boundary between "close" and "far":

Choosing m:

Numerical Examples

Let margin m = 2.0.

Example 1: Similar pair, close

Example 2: Similar pair, far

Example 3: Dissimilar pair, close

Example 4: Dissimilar pair, far

The Gradient

For similar pairs (y = 0):

\frac{\partial L}{\partial e_1} = (e_1 - e_2) Gradient points from e2 toward e1, so e1 moves toward e2.

For dissimilar pairs (y = 1) when D < m:

\frac{\partial L}{\partial e_1} = -(m - D) \cdot \frac{e_1 - e_2}{D} Gradient points in the opposite direction, pushing e1 away from e2.

For dissimilar pairs when D >= m:

\frac{\partial L}{\partial e_1} = 0 No gradient. The pair is already sufficiently separated.

Creating Training Pairs

Contrastive loss requires pairs of samples. Strategies:

From labeled data:

Self-supervised (no labels):

Mining strategies:

The Embedding Space

Well-trained embeddings form clusters:

Properties of the learned space:

Contrastive vs. Triplet Loss

Contrastive loss works with pairs. Triplet loss works with triplets (anchor, positive, negative).

Contrastive:

Triplet:

Both achieve similar goals; triplet loss is often considered more sample efficient.

Where Contrastive Loss Is Used

Examples

Example 1

Input
a = [1.0, 0.0], b = [1.0, 0.0], y = [1], margin = 1.0, reduction = "mean"
Output
0.0
Explanation
The similar embeddings have zero distance, so their squared-distance loss is zero.

Example 2

Input
a = [0.0, 0.0], b = [0.5, 0.0], y = [0], margin = 1.0, reduction = "mean"
Output
0.25

Example 3

Input
a = [[0.0, 0.0], [1.0, 1.0]], b = [[0.0, 0.0], [2.0, 2.0]], y = [1, 0], margin = 1.0, reduction = "mean"
Output
0.0

Hints

  1. Convert a one-dimensional difference to shape (1, D) before reducing over axis 1.
  2. Use np.linalg.norm(a - b, axis=1) for the pairwise distances.
  3. Build the positive and negative terms separately before applying the reduction.

Requirements

Constraints

Starter Code

import numpy as np

def contrastive_loss(a: list, b: list, y: list, margin: float = 1.0, reduction: str = "mean") -> float:
    """
    Returns the loss as a float.
    """
    # Write code here
    pass

Test Cases

CaseMatches
Identical positive pair—public
Dissimilar pair within margin—public
Batch with mixed labels—public