পাঠ ২৫ · ৩৩-এর মধ্যে · মডিউল ৪

Home / AI Courses / MLOps / Data drift

Data drift detection — PSI, KS

Data drift — statistical tests

৭ মিনিট পড়া মধ্য · Intermediate Stats

এই পাঠে যা শিখবেন

PSI computation + thresholds
KS test — when, how
Chi-square for categorical
False positive control — window size, seasonality

১ · Drift types

Covariate drift / data drift: $P(X)$ changes; $P(y|X)$ same.
Concept drift: $P(y|X)$ changes (Lesson 26).
Label drift / prior shift: $P(y)$ changes.

২ · PSI — Population Stability Index

$$ \text{PSI} = \sum_i (a_i - b_i) \ln\left(\frac{a_i}{b_i}\right) $$

যেখানে $a_i$ = baseline-এ bin $i$ এর fraction, $b_i$ = current-এ bin $i$।

Thresholds:

PSI < 0.1 — no significant change.
0.1 ≤ PSI < 0.25 — moderate, watch.
PSI ≥ 0.25 — significant, alert.

৩ · PSI Python implementation

Python · PSI

import numpy as np
import pandas as pd

def psi(baseline: np.ndarray, current: np.ndarray, n_bins: int = 10) -> float:
    """Compute Population Stability Index."""
    # Bin edges from baseline (quantile-based)
    breaks = np.unique(np.quantile(baseline, np.linspace(0, 1, n_bins + 1)))
    if len(breaks) < 2:
        return 0.0

    base_counts = np.histogram(baseline, bins=breaks)[0].astype(float)
    curr_counts = np.histogram(current, bins=breaks)[0].astype(float)

    # Add small epsilon — avoid log(0)
    eps = 1e-6
    base_pct = (base_counts + eps) / (base_counts.sum() + eps * len(base_counts))
    curr_pct = (curr_counts + eps) / (curr_counts.sum() + eps * len(curr_counts))

    return float(((curr_pct - base_pct) * np.log(curr_pct / base_pct)).sum())


# Usage
baseline_age = np.random.normal(35, 10, 10_000)        # training data
current_age = np.random.normal(38, 12, 5_000)          # last 7 days production

p = psi(baseline_age, current_age)
print(f"PSI: {p:.3f}")
if p > 0.25:
    print("⚠️ Significant drift")
elif p > 0.1:
    print("👀 Moderate drift, watch")
else:
    print("✓ Stable")

Quantile-based bins data-driven; equal-frequency. Categorical-এর জন্য each unique value bin. Run on schedule (hourly/daily) per feature.

৪ · KS test — Kolmogorov-Smirnov

Continuous distribution-এর CDF-difference.

$$ D = \sup_x |F_1(x) - F_2(x)| $$

scipy ks_2samp — direct.
p-value < 0.05 — significantly different.
Sensitivity to large samples — even small drift "significant" statistically.

৫ · Chi-square — categorical

Categorical feature distribution compare.

$\chi^2 = \sum (O - E)^2 / E$।
Each unique category — observed vs expected.
scipy chi2_contingency.

৬ · False positive control

Window size: too small → noisy; too large → slow detection. Typical: 7-day window.
Seasonality: Friday vs Monday traffic ভিন্ন। Compare same-day-of-week.
Multiple comparisons: 50 features × daily check = ~2.5 false alarm/day at p=0.05. Bonferroni correction.
Practical: PSI threshold prefer over p-value (less sample-size sensitive).

৭ · Drift types — covariate vs label

Covariate drift sole-এ accuracy decay না হতে পারে — যদি conditional $P(y|X)$ same থাকে। Concept drift এর সাথে কম্বাইনেশনে damaging.

Important features-এ drift — high-impact.
Low-importance features-এ drift — sometimes ignorable.
Feature importance ranking — multi-feature drift prioritization.

PSI thresholds — industry-standard 0.1 ও 0.25. Banking-এ stricter; experimental ML-এ looser acceptable.

Bangladesh-এ Bangla NLP feature drift common — slang evolution, dialect shift, social topic change. Quarterly retraining without drift monitoring even-then necessary.

ভাবনার প্রশ্ন

প্র ০১"Seasonality vs drift — distinguish কীভাবে?"

Seasonal pattern drift মতো দেখায় but recurring.

Examples:

Daraz Eid sale — purchase distribution massive shift, expected.
Friday Jumma — Pathao demand spike আসে।
Monsoon — food delivery surge.

Strategies:

Compare year-over-year: Eid 2025 vs Eid 2024 — drift if বদল।
Compare same-day-of-week: Friday vs last Friday.
Detrending: remove seasonal component; check residual drift.
Multiple baselines: per-season baseline maintain.
Calendar-aware threshold: Eid week relax PSI threshold.

BD calendar-specific:

Eid (2× yearly), Pohela Boishakh, Victory Day, Independence Day, Durga Puja, Christmas.
Each has distinct distribution. Calendar-aware monitoring built-in.

মূল উপলব্ধি: Naive drift detection seasonal-tag confused. Calendar-context essential. BD-specific events care.

প্র ০২"Window size choice — 1 day, 7 day, 30 day?"

Window size = sensitivity vs noise trade-off.

1 day:

Pros: fast detection.
Cons: noisy, weekly variation captured.
Use: very sensitive features (fraud signal).

7 day:

Pros: weekly cycle averaged, less noise.
Cons: 1-week lag.
Use: most production cases — sweet spot.

30 day:

Pros: very stable.
Cons: slow detection.
Use: long-term trend trackingে।

Multiple windows:

1-day for quick alert.
7-day for confirmed trend.
30-day for trend dashboard.

Sliding vs tumbling:

Sliding: rolling 7 days, daily compute.
Tumbling: discrete weeks Sat-Fri.
Sliding more common, smoother.

মূল উপলব্ধি: Default 7-day sliding. Sensitive add 1-day. Strategic add 30-day. Single window не покрывает all need.

প্র ০৩"Covariate drift detected, কিন্তু accuracy ঠিক — retrain করব?"

Common confusion.

Scenario:

Feature distribution shifted.
Ground truth available — accuracy stable.
Should retrain?

Considerations:

Don't retrain:

Accuracy stable — model robust to shift.
Drift may revert (seasonal).
Retraining cost > benefit.

Do retrain:

Drift in important feature consistently.
Accuracy stable but borderline.
"Future-proofing" — drift growing.
New data improves model fundamentally.

Better question:

"Is drift growing or stabilizing?"
"Has subgroup performance shifted?" (overall stable can hide).
"Concept drift might follow covariate drift."

Cost analysis:

Retraining cost (compute + engineer).
Risk of new model regression.
"Don't fix what works".

Bangladesh fintech reality:

Quarterly scheduled retrain — drift handled by routine.
Mid-quarter drift alert — investigate, retrain only if pattern persists.

মূল উপলব্ধি: Drift alert ≠ automatic retrain trigger. Investigate first. Retraining-this is risk-reward decision. Routine schedule + emergency retrain hybrid.

প্র ০৪"50 feature daily PSI = noisy — practical multi-feature alerting?"

Multiple comparisons issue.

Problem:

50 features × daily check = many alerts.
Some "drift" naturally; alert fatigue.

Strategies:

(১) Feature importance weight:

Top-10 features only alert; rest digest.
SHAP-based importance from training.

(২) Aggregate metric:

"Number of features in drift" daily count.
Sudden spike alert.

(৩) Bonferroni correction:

α = 0.05 / 50 = 0.001 effective.
Stricter threshold; fewer false positives.

(৪) Hierarchical alert:

Single critical feature drift → page.
3+ feature drift → Slack.
Less critical → daily digest report.

(৫) Concept-drift-only alert:

Covariate drift digest only.
Concept drift (accuracy drop) → escalate.

BD context:

Mid-size fintech 30-50 features common.
Top-10 + aggregate strategy practical.

মূল উপলব্ধি: Feature drift = pruning + prioritization problem. All-feature alert noise; top-feature + aggregate signal-to-noise ভাল।

অনুশীলন

Compute: উপরের PSI function নিজের sample data-তে চালান। Drift inject করুন, threshold trigger.
Baseline mean=35; current mean=45 — PSI ~0.3-0.5. Threshold trigger.
KS test: scipy.stats.ks_2samp ব্যবহার করুন; PSI-এর সাথে compare.
from scipy.stats import ks_2samp; stat, p = ks_2samp(baseline, current). Same drift PSI ও KS দু'টোতেই ধরা পড়ে।
চিন্তা: Pathao surge model — top 5 features কী হবে; PSI threshold কী set করবেন?
- Driver count in zone (last hour) — PSI > 0.2 alert (sensitive).
- Avg ride duration — PSI > 0.25 alert.
- Time of day (categorical) — Chi-square seasonal aware.
- Weather — PSI > 0.3 (high seasonality natural).
- Day of week — exclude (seasonality dominant).