পাঠ ২৫ · ৩৩-এর মধ্যে · মডিউল ৪
Home / AI Courses / MLOps / Data drift

Data drift detection — PSI, KS

Data drift — statistical tests
৭ মিনিট পড়া মধ্য · Intermediate Stats

এই পাঠে যা শিখবেন

  • PSI computation + thresholds
  • KS test — when, how
  • Chi-square for categorical
  • False positive control — window size, seasonality

১ · Drift types

  • Covariate drift / data drift: $P(X)$ changes; $P(y|X)$ same।
  • Concept drift: $P(y|X)$ changes (Lesson 26)।
  • Label drift / prior shift: $P(y)$ changes।

২ · PSI — Population Stability Index

$$ \text{PSI} = \sum_i (a_i - b_i) \ln\left(\frac{a_i}{b_i}\right) $$

যেখানে $a_i$ = baseline-এ bin $i$ এর fraction, $b_i$ = current-এ bin $i$।

Thresholds:

  • PSI < 0.1 — no significant change।
  • 0.1 ≤ PSI < 0.25 — moderate, watch।
  • PSI ≥ 0.25 — significant, alert।

৩ · PSI Python implementation

Python · PSI 
import numpy as np
import pandas as pd

def psi(baseline: np.ndarray, current: np.ndarray, n_bins: int = 10) -> float:
    """Compute Population Stability Index."""
    # Bin edges from baseline (quantile-based)
    breaks = np.unique(np.quantile(baseline, np.linspace(0, 1, n_bins + 1)))
    if len(breaks) < 2:
        return 0.0

    base_counts = np.histogram(baseline, bins=breaks)[0].astype(float)
    curr_counts = np.histogram(current, bins=breaks)[0].astype(float)

    # Add small epsilon — avoid log(0)
    eps = 1e-6
    base_pct = (base_counts + eps) / (base_counts.sum() + eps * len(base_counts))
    curr_pct = (curr_counts + eps) / (curr_counts.sum() + eps * len(curr_counts))

    return float(((curr_pct - base_pct) * np.log(curr_pct / base_pct)).sum())


# Usage
baseline_age = np.random.normal(35, 10, 10_000)        # training data
current_age = np.random.normal(38, 12, 5_000)          # last 7 days production

p = psi(baseline_age, current_age)
print(f"PSI: {p:.3f}")
if p > 0.25:
    print("⚠️ Significant drift")
elif p > 0.1:
    print("👀 Moderate drift, watch")
else:
    print("✓ Stable")

    


    

      Quantile-based bins data-driven; equal-frequency। Categorical-এর জন্য each unique value bin। Run on schedule (hourly/daily) per feature।
    


    
৪ · KS test — Kolmogorov-Smirnov


    

      Continuous distribution-এর CDF-difference।
    

    

      $$ D = \sup_x |F_1(x) - F_2(x)| $$
    


    
    
      
  • scipy ks_2samp — direct।
    • 
      
  • p-value < 0.05 — significantly different।
    • 
      
  • Sensitivity to large samples — even small drift "significant" statistically।
    • 
    


    
৫ · Chi-square — categorical


    

      Categorical feature distribution compare।
    

    
    
      
  • $\chi^2 = \sum (O - E)^2 / E$।
    • 
      
  • Each unique category — observed vs expected।
    • 
      
  • scipy chi2_contingency।
    • 
    


    
৬ · False positive control


    
    
      
  • Window size: too small → noisy; too large → slow detection। Typical: 7-day window।
    • 
      
  • Seasonality: Friday vs Monday traffic ভিন্ন। Compare same-day-of-week।
    • 
      
  • Multiple comparisons: 50 features × daily check = ~2.5 false alarm/day at p=0.05। Bonferroni correction।
    • 
      
  • Practical: PSI threshold prefer over p-value (less sample-size sensitive)।
    • 
    


    
৭ · Drift types — covariate vs label


    

      Covariate drift sole-এ accuracy decay না হতে পারে — যদি conditional $P(y|X)$ same থাকে। Concept drift এর সাথে কম্বাইনেশনে damaging।
    

    
    
      
  • Important features-এ drift — high-impact।
    • 
      
  • Low-importance features-এ drift — sometimes ignorable।
    • 
      
  • Feature importance ranking — multi-feature drift prioritization।
    • 
    


    

      
        
        PSI thresholds — drift severity

        
        
        

        0
        0.1
        0.25
        ∞

        
        

        
        Stable
        no action

        
        Watch
        investigate, prepare

        
        Significant drift
        alert, retraining trigger

        Per-feature PSI computed daily — top features-এ alert critical।
        "Stable" baseline = training data; "current" = recent production।
      
      
PSI thresholds — industry-standard 0.1 ও 0.25। Banking-এ stricter; experimental ML-এ looser acceptable।

    


    

      Bangladesh-এ Bangla NLP feature drift common — slang evolution, dialect shift, social topic change। Quarterly retraining without drift monitoring even-then necessary।
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      প্র ০১"Seasonality vs drift — distinguish কীভাবে?"
      

        
Seasonal pattern drift মতো দেখায় but recurring।

        
Examples:

        
    
          
  • Daraz Eid sale — purchase distribution massive shift, expected।
    • 
          
  • Friday Jumma — Pathao demand spike আসে।
    • 
          
  • Monsoon — food delivery surge।
    • 
        

        
Strategies:

        
    
          
  • Compare year-over-year: Eid 2025 vs Eid 2024 — drift if বদল।
    • 
          
  • Compare same-day-of-week: Friday vs last Friday।
    • 
          
  • Detrending: remove seasonal component; check residual drift।
    • 
          
  • Multiple baselines: per-season baseline maintain।
    • 
          
  • Calendar-aware threshold: Eid week relax PSI threshold।
    • 
        

        
BD calendar-specific:

        
    
          
  • Eid (2× yearly), Pohela Boishakh, Victory Day, Independence Day, Durga Puja, Christmas।
    • 
          
  • Each has distinct distribution। Calendar-aware monitoring built-in।
    • 
        

        
মূল উপলব্ধি: Naive drift detection seasonal-tag confused। Calendar-context essential। BD-specific events care।

      

    


    

      প্র ০২"Window size choice — 1 day, 7 day, 30 day?"
      

        
Window size = sensitivity vs noise trade-off।

        
1 day:

        
    
          
  • Pros: fast detection।
    • 
          
  • Cons: noisy, weekly variation captured।
    • 
          
  • Use: very sensitive features (fraud signal)।
    • 
        

        
7 day:

        
    
          
  • Pros: weekly cycle averaged, less noise।
    • 
          
  • Cons: 1-week lag।
    • 
          
  • Use: most production cases — sweet spot।
    • 
        

        
30 day:

        
    
          
  • Pros: very stable।
    • 
          
  • Cons: slow detection।
    • 
          
  • Use: long-term trend trackingে।
    • 
        

        
Multiple windows:

        
    
          
  • 1-day for quick alert।
    • 
          
  • 7-day for confirmed trend।
    • 
          
  • 30-day for trend dashboard।
    • 
        

        
Sliding vs tumbling:

        
    
          
  • Sliding: rolling 7 days, daily compute।
    • 
          
  • Tumbling: discrete weeks Sat-Fri।
    • 
          
  • Sliding more common, smoother।
    • 
        

        
মূল উপলব্ধি: Default 7-day sliding। Sensitive add 1-day। Strategic add 30-day। Single window не покрывает all need।

      

    


    

      প্র ০৩"Covariate drift detected, কিন্তু accuracy ঠিক — retrain করব?"
      

        
Common confusion।

        
Scenario:

        
    
          
  • Feature distribution shifted।
    • 
          
  • Ground truth available — accuracy stable।
    • 
          
  • Should retrain?
    • 
        

        
Considerations:

        
Don't retrain:

        
    
          
  • Accuracy stable — model robust to shift।
    • 
          
  • Drift may revert (seasonal)।
    • 
          
  • Retraining cost > benefit।
    • 
        

        
Do retrain:

        
    
          
  • Drift in important feature consistently।
    • 
          
  • Accuracy stable but borderline।
    • 
          
  • "Future-proofing" — drift growing।
    • 
          
  • New data improves model fundamentally।
    • 
        

        
Better question:

        
    
          
  • "Is drift growing or stabilizing?"
    • 
          
  • "Has subgroup performance shifted?" (overall stable can hide)।
    • 
          
  • "Concept drift might follow covariate drift."
    • 
        

        
Cost analysis:

        
    
          
  • Retraining cost (compute + engineer)।
    • 
          
  • Risk of new model regression।
    • 
          
  • "Don't fix what works"।
    • 
        

        
Bangladesh fintech reality:

        
    
          
  • Quarterly scheduled retrain — drift handled by routine।
    • 
          
  • Mid-quarter drift alert — investigate, retrain only if pattern persists।
    • 
        

        
মূল উপলব্ধি: Drift alert ≠ automatic retrain trigger। Investigate first। Retraining-this is risk-reward decision। Routine schedule + emergency retrain hybrid।

      

    


    

      প্র ০৪"50 feature daily PSI = noisy — practical multi-feature alerting?"
      

        
Multiple comparisons issue।

        
Problem:

        
    
          
  • 50 features × daily check = many alerts।
    • 
          
  • Some "drift" naturally; alert fatigue।
    • 
        

        
Strategies:

        
(১) Feature importance weight:

        
    
          
  • Top-10 features only alert; rest digest।
    • 
          
  • SHAP-based importance from training।
    • 
        

        
(২) Aggregate metric:

        
    
          
  • "Number of features in drift" daily count।
    • 
          
  • Sudden spike alert।
    • 
        

        
(৩) Bonferroni correction:

        
    
          
  • α = 0.05 / 50 = 0.001 effective।
    • 
          
  • Stricter threshold; fewer false positives।
    • 
        

        
(৪) Hierarchical alert:

        
    
          
  • Single critical feature drift → page।
    • 
          
  • 3+ feature drift → Slack।
    • 
          
  • Less critical → daily digest report।
    • 
        

        
(৫) Concept-drift-only alert:

        
    
          
  • Covariate drift digest only।
    • 
          
  • Concept drift (accuracy drop) → escalate।
    • 
        

        
BD context:

        
    
          
  • Mid-size fintech 30-50 features common।
    • 
          
  • Top-10 + aggregate strategy practical।
    • 
        

        
মূল উপলব্ধি: Feature drift = pruning + prioritization problem। All-feature alert noise; top-feature + aggregate signal-to-noise ভাল।

      

    

  


  

    
 অনুশীলন

    
    
      
  1. Compute: উপরের PSI function নিজের sample data-তে চালান। Drift inject করুন, threshold trigger।
        
    Baseline mean=35; current mean=45 — PSI ~0.3-0.5। Threshold trigger।
    
      
    2. 
      
  2. KS test: scipy.stats.ks_2samp ব্যবহার করুন; PSI-এর সাথে compare।
        
    from scipy.stats import ks_2samp; stat, p = ks_2samp(baseline, current)। Same drift PSI ও KS দু'টোতেই ধরা পড়ে।
    
      
    3. 
      
  3. চিন্তা: Pathao surge model — top 5 features কী হবে; PSI threshold কী set করবেন?
        
    
          
      
            
    • Driver count in zone (last hour) — PSI > 0.2 alert (sensitive)।
      • 
            
    • Avg ride duration — PSI > 0.25 alert।
      • 
            
    • Time of day (categorical) — Chi-square seasonal aware।
      • 
            
    • Weather — PSI > 0.3 (high seasonality natural)।
      • 
            
    • Day of week — exclude (seasonality dominant)।
      • 
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন

    
  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ২৪ · Model monitoring