AI Engineer Exhaustive
EasyMLOps

Data Drift Detection

MLOps

Easy

Problem

Compare a production histogram with a reference histogram using total variation distance. First normalize each list of counts into a probability distribution. Then compute:

D_{TV}(P,Q)=\frac{1}{2}\sum_i |p_i-q_i|

Here, p_i and q_i are the normalized probabilities for bin i. Drift is detected only when the score is strictly greater than the threshold. Return a dictionary containing the score as a float and the drift_detected flag as a Boolean.

Theory

Machine learning models are trained on historical data. They learn patterns that existed at a specific point in time. But the world does not stand still. User behaviors evolve, market conditions shift, seasonal patterns emerge, and the underlying data distribution changes. When the data a model encounters in production differs significantly from its training data, the model performance degrades. This phenomenon is called data drift.

Data drift is insidious because it happens silently. Unlike a system crash or an error message, drift causes predictions to become gradually less accurate without any obvious warning. By the time you notice the degraded business metrics, the damage may already be significant.

Consider a few examples:

Proactive drift detection allows teams to identify distribution changes early and take corrective action before model performance severely degrades.

Types of Drift

Before diving into detection methods, it is important to understand the different types of drift:

Covariate Drift (Feature Drift): The distribution of input features changes, but the relationship between features and target remains the same. For example, if your user base shifts from mostly young users to mostly older users, the age distribution changes, but age may still predict behavior the same way.

Prior Probability Drift (Label Drift): The distribution of the target variable changes. For example, if fraud rates increase from 1% to 5%, models trained on the old base rate may underperform.

Concept Drift: The relationship between features and target changes. A feature that was predictive may become less so, or the prediction boundary may shift. This is the most challenging type because even if distributions look similar, the underlying patterns have changed.

This problem focuses on covariate drift detection by comparing feature distributions between reference (training) and production data.

Measuring Drift with Total Variation Distance

Multiple statistical measures can quantify distribution differences. The Total Variation Distance (TVD) is one of the simplest and most intuitive metrics.

TVD measures the maximum difference between two probability distributions. For discrete distributions represented as histograms, it sums the absolute differences across all bins and divides by two.

The factor of 1/2 normalizes the metric so that TVD ranges from 0 to 1:

The Mathematical Definition

Given two probability distributions P (reference) and Q (production), each represented as a histogram with n bins, the Total Variation Distance is:

\text{TVD}(P, Q) = \frac{1}{2} \sum_{i=1}^{n} |p_i - q_i|

where p_i is the probability mass in bin i for distribution P, and q_i is the probability mass in bin i for distribution Q.

Key Properties of TVD:

From Raw Counts to Probability Distributions

In practice, you work with histogram bin counts rather than probabilities. Before computing TVD, you must normalize these counts to probability distributions.

Given reference counts [c_1^{ref}, c_2^{ref}, ..., c_n^{ref}] and production counts [c_1^{prod}, c_2^{prod}, ..., c_n^{prod}]:

Step 1: Compute totals

N_{ref} = \sum_{i=1}^{n} c_i^{ref} \quad \text{and} \quad N_{prod} = \sum_{i=1}^{n} c_i^{prod}

Step 2: Normalize to probabilities

p_i = \frac{c_i^{ref}}{N_{ref}} \quad \text{and} \quad q_i = \frac{c_i^{prod}}{N_{prod}}

This normalization ensures that each distribution sums to 1, regardless of how many samples are in each dataset.

Step-by-Step TVD Computation

Given reference histogram counts and production histogram counts:

Step 1: Sum all reference counts to get the total: N_{ref}

Step 2: Divide each reference count by the total to get the reference probability distribution

Step 3: Sum all production counts to get the total: N_{prod}

Step 4: Divide each production count by the total to get the production probability distribution

Step 5: For each bin i, compute the absolute difference: |p_i - q_i|

Step 6: Sum all absolute differences

Step 7: Divide by 2 to get the final TVD score

Step 8: Compare TVD against the threshold to determine if drift has occurred

Setting the Drift Threshold

The choice of threshold depends on your context and tolerance for false alarms:

The drift decision is:

\text{drift\_detected} = (\text{TVD} > \text{threshold})

Note the strict inequality. A TVD exactly equal to the threshold is not considered drift.

A Detailed Worked Example

Suppose you are monitoring a feature called "purchase_amount" discretized into 5 bins representing price ranges.

Reference Data (from training period):

Total reference samples: 10,000

Reference probabilities:

Production Data (recent week):

Total production samples: 8,000

Production probabilities:

Computing TVD:

Sum of absolute differences: 0.00 + 0.05 + 0.05 + 0.10 + 0.00 = 0.20

TVD: 0.20 / 2 = 0.10

Interpretation: With a threshold of 0.10, this feature is right at the boundary. Since drift is detected only when TVD is strictly greater than the threshold, drift would NOT be detected in this case. However, this warrants close monitoring as the feature is showing moderate shift.

Looking at the data, we can see that production has shifted toward higher purchase amounts (Bin 4 increased from 20% to 30%), which might indicate a change in customer behavior or product mix.

Comparison with Other Drift Metrics

TVD is not the only way to measure distribution differences. Here are some alternatives:

Kullback-Leibler (KL) Divergence: Measures information loss when approximating one distribution with another. Asymmetric and can be infinite if distributions do not overlap.

Jensen-Shannon (JS) Divergence: Symmetric version of KL divergence, bounded between 0 and 1. Popular in ML applications.

Population Stability Index (PSI): Common in financial modeling, similar structure to KL divergence but symmetric.

Kolmogorov-Smirnov (KS) Test: Measures maximum difference between cumulative distribution functions. Provides statistical significance.

Wasserstein Distance (Earth Mover's Distance): Measures "cost" of transforming one distribution to another. Useful when bin ordering matters.

TVD's advantages are simplicity, interpretability, and bounded range. Its main limitation is that it treats all bin differences equally regardless of which bins shifted.

Practical Considerations for Drift Detection

Choosing Bins: The number and boundaries of histogram bins affect drift sensitivity. Too few bins may miss subtle shifts; too many may create noise. Common approaches include equal-width bins, equal-frequency bins, or domain-specific boundaries.

Sample Size: Drift detection requires sufficient samples in both reference and production datasets. With small samples, random variation can masquerade as drift.

Reference Period: Decide whether your reference is a fixed historical snapshot or a rolling window. Fixed references detect drift from a baseline; rolling windows detect recent changes.

Multiple Features: Real systems monitor many features. You may need to correct for multiple comparisons or aggregate drift scores across features.

Alerting Strategy: Decide whether to alert immediately when drift is detected or wait for drift to persist across multiple time windows to reduce false alarms.

Where Data Drift Detection Shows Up

ML Model Monitoring: Production ML platforms continuously compare incoming feature distributions against baselines. Drift alerts trigger investigations and potential retraining.

Data Quality Monitoring: Data engineering teams use drift detection to catch upstream changes in data pipelines before they propagate to downstream consumers.

A/B Testing Validation: Before analyzing experiment results, teams verify that test and control groups have similar baseline feature distributions.

Regulatory Compliance: Financial institutions must demonstrate that models remain valid on current populations, requiring regular distribution comparisons.

Predictive Maintenance: Manufacturing systems detect when sensor readings drift from normal operating ranges, potentially indicating equipment degradation.

Examples

Example 1

Input
reference_counts = [50, 50], production_counts = [55, 45], threshold = 0.1
Output
{"score": 0.05, "drift_detected": false}
Explanation
The absolute probability differences sum to 0.1, so TVD is 0.05.

Example 2

Input
reference_counts = [50, 50], production_counts = [90, 10], threshold = 0.1
Output
{"score": 0.4, "drift_detected": true}

Hints

  1. Divide every bin count by the total count of its histogram.
  2. Sum the absolute differences between matching probabilities, then multiply by one half.

Requirements

Constraints

Starter Code

def detect_drift(reference_counts: list, production_counts: list, threshold: float) -> dict:
    """
    Returns a dictionary with score and drift_detected.
    """
    # Write code here
    pass

Test Cases

CaseMatches
no drift—public
drift detected—public