পাঠ ২৩ · ৩৩-এর মধ্যে · মডিউল ৩
Home / AI Courses / MLOps / A/B testing

A/B testing for ML

A/B testing — hypothesis to decision
৭ মিনিট পড়া উচ্চ · Advanced Stats

এই পাঠে যা শিখবেন

  • Hypothesis + MDE + sample size calculation
  • Primary vs guardrail metrics
  • Pitfalls — peeking, novelty, attribution
  • Multi-armed bandit alternative

১ · A/B vs Canary

Canary = deployment safety mechanism (does new version break?)। A/B test = product/business decision (does new version improve metrics?)।

  • Canary: short, infrastructure metric (latency, error)।
  • A/B: long (days/weeks), business metric (conversion, revenue)।
  • Canary first, then A/B-এ promote।

২ · Hypothesis design

একটি good hypothesis:

  • Null (H0): new model = control। No improvement।
  • Alternate (H1): new model conversion +5%।
  • Primary metric: conversion rate।
  • Guardrails: latency < baseline+20ms; error rate < baseline।
  • Population: all users in BD region।
  • Duration: 14 days (capture weekly cycle)।

৩ · MDE ও sample size

MDEMinimum Detectable Effectsmallest improvement test can statistically detect with given power। Smaller MDE → bigger sample size। = "smallest improvement we care about"।

$$ n \approx \frac{2 \sigma^2 (z_{\alpha/2} + z_\beta)^2}{\text{MDE}^2} $$

Per arm sample size। For binary metric (conversion):

$$ n \approx \frac{2 p (1-p) (z_{\alpha/2} + z_\beta)^2}{\text{MDE}^2} $$

৪ · Sample size calculation

Python · Sample size 
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

# Current conversion 8%
# Want to detect 5% relative improvement → 8% → 8.4%
baseline = 0.08
mde_relative = 0.05
treatment = baseline * (1 + mde_relative)

effect = proportion_effectsize(baseline, treatment)
analysis = NormalIndPower()
n_per_arm = analysis.solve_power(
    effect_size=effect,
    alpha=0.05,
    power=0.80,
    alternative="two-sided",
)
print(f"Per arm sample size: {int(n_per_arm):,}")

# At 1000 users/day per arm = days_needed
total_traffic = 50_000  # daily users
split = 0.5  # 50/50
daily_per_arm = total_traffic * split
days = n_per_arm / daily_per_arm
print(f"Duration: ~{days:.1f} days")

    


    

      Realistic numbers: 8% conversion + 5% relative MDE → ~80,000 per arm needed। At 25,000/day → ~3 days। Power 80%, α 5% standard।
    


    
৫ · Primary vs guardrail


    
    
      
  • Primary: the metric we want to improve।
    • 
      
  • Guardrails: shouldn't regress।
    • 
    


    

      Recommendation example:
    

    
    
      
  • Primary: click-through rate।
    • 
      
  • Guardrails: page load latency, bounce rate, complaint rate।
    • 
    


    
৬ · Common pitfalls


    
    
      
  • Peeking: daily check + early stop on win = false positive। Pre-register decision time।
    • 
      
  • Sample ratio mismatch (SRM): 50/50 expected, 49/51 observed — assignment bug? Check chi-square।
    • 
      
  • Novelty effect: new feature attention boost initial; fades। Run longer।
    • 
      
  • Carry-over: user-level randomization দরকার, not session।
    • 
      
  • Network effects: social product — control + treatment user-influence each other।
    • 
    


    
৭ · Multi-armed bandit alternative


    

      MAB = adaptive allocation। Better-performing arm → more traffic। Reduce regret।
    

    
    
      
  • Pros: faster convergence, less wasted traffic on bad variants।
    • 
      
  • Cons: harder analysis, statistical significance complex।
    • 
      
  • Use when: many variants, traffic limited, "maximize during test"।
    • 
      
  • Skip when: classical A/B preference (regulator-friendly)।
    • 
    


    
৮ · ML metric vs business metric


    

      Often divergent।
    

    
    
      
  • New model AUC +2% (offline) → CTR +0% (online)।
    • 
      
  • Reasons: offline test ≠ online distribution; user behavior change।
    • 
      
  • Always trust online business metric over offline ML metric।
    • 
    


    

      
        
        A/B test workflow — design → run → decide

        
        Design
        hypothesis, MDE

        
        Sample size
        power calc

        
        Randomize
        user-level hash

        
        Run experiment
        2 weeks

        
        SRM check
        assignment OK?

        
        Compute metrics
        primary + guardrail

        
        Stat test
        CI, p-value

        
        Decision
        launch / hold / kill

        
        
        
        
        
        
        

        Pre-register decision criteria avoid post-hoc rationalization।
      
      
A/B test workflow — design statistically up-front; run end-to-end; decision criteria pre-registered।

    


    

      Bangladesh-এ A/B test platform — Daraz, Pathao, bKash, Foodpanda — সবাই custom বা Optimizely-style external use।
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      প্র ০১"ML metric +5% but business metric flat — কেন? কী করবেন?"
      

        
Common ML productivity paradox।

        
Why divergence:

        
    
          
  • Offline test set distribution ≠ online production distribution।
    • 
          
  • Selection bias — recommendation only logged for shown items।
    • 
          
  • User behavior adaptation — better recommendation, but user not click more।
    • 
          
  • Confounding — recommendation only one of many factors।
    • 
          
  • Saturation — user already at maximum daily clicks।
    • 
        

        
Investigation steps:

        
    
          
  • Subgroup analysis — maybe wins concentrate in some segments।
    • 
          
  • Time-of-day analysis — peak vs off-peak ভিন্ন।
    • 
          
  • Funnel — click increased but conversion not?
    • 
          
  • Counterfactual — "if old model would have shown different items"।
    • 
        

        
Action options:

        
    
          
  • Hold — A/B test data not strong enough; extend duration।
    • 
          
  • Ship anyway — ML metric improvement long-term value।
    • 
          
  • Iterate — discovered offline-online gap; fix offline metric.
    • 
          
  • Kill — ROI not justifying complexity।
    • 
        

        
BD context — Daraz example:

        
    
          
  • NDCG +5% offline → CTR flat online।
    • 
          
  • Investigation: Daraz users prefer cheap items; new model more diverse but pricier।
    • 
          
  • Fix: business-aware loss function — price × CTR optimize।
    • 
        

        
মূল উপলব্ধি: Offline-online gap normal। Trust online business metric। Use offline for sanity-check, online for ship decision।

      

    


    

      প্র ০২"Long-term effects — A/B test 2 weeks, but real impact 6 months পরে?"
      

        
Long-term effect কঠিন measure কিন্তু critical।

        
Examples:

        
    
          
  • Aggressive engagement model → short-term DAU up → long-term burnout।
    • 
          
  • Push notification volume → click-through up → unsubscribe later।
    • 
          
  • Aggressive ad placement → revenue up → user trust down।
    • 
        

        
Strategies:

        
(১) Holdout group long-term:

        
    
          
  • 5% users never see treatment, even after launch।
    • 
          
  • 6 month after — compare retention, lifetime value।
    • 
          
  • Cost: small group continuously held back।
    • 
        

        
(২) Surrogate metrics:

        
    
          
  • Short-term proxies for long-term।
    • 
          
  • e.g., "user who clicks unsubscribe in week 1 = lost user month 6"।
    • 
          
  • Validate proxy initially with retrospective।
    • 
        

        
(৩) Phased rollout with monitoring:

        
    
          
  • Launch but continue monitoring 3-6 months।
    • 
          
  • Long-term metric degrade → rollback।
    • 
        

        
(৪) Switchback experiments:

        
    
          
  • Toggle on/off weekly — measure lift each cycle।
    • 
          
  • Long-term carryover insight।
    • 
        

        
BD context:

        
    
          
  • Bangladesh subscription business — retention 6-12 month most important।
    • 
          
  • Holdout group + surrogate combination।
    • 
        

        
মূল উপলব্ধি: Long-term measurement structural commitment। Holdout permanent, surrogate validated, post-launch monitoring continued। Skip — eventually quality degrades silently।

      

    


    

      প্র ০৩"Network effects — social product-এ A/B testing চ্যালেঞ্জ।"
      

        
Social/network products — control + treatment interact। Standard A/B violates assumption।

        
Examples:

        
    
          
  • "Better feed algorithm" → user post more → control's feed change-ও।
    • 
          
  • Marketplace — better seller experience → buyers see better products → control buyers benefit।
    • 
        

        
Solutions:

        
(১) Cluster randomization:

        
    
          
  • Assign by geography (city, region)।
    • 
          
  • Within cluster, single condition — interactions stay within।
    • 
          
  • Trade-off: bigger sample needed; fewer "users" for statistical purposes।
    • 
        

        
(২) Time-based switchback:

        
    
          
  • Whole platform alternates control/treatment per hour or day।
    • 
          
  • Used by Uber, Lyft (matching algorithm)।
    • 
          
  • Requires no carryover-effects।
    • 
        

        
(৩) Ego-network analysis:

        
    
          
  • Friends-of-friends control।
    • 
          
  • Account treatment effect on neighbors।
    • 
        

        
BD context:

        
    
          
  • Pathao — driver-passenger marketplace; cluster by city।
    • 
          
  • Daraz — seller-buyer; product category cluster।
    • 
        

        
মূল উপলব্ধি: Network products — standard A/B mismeasures। Cluster or switchback essential। Statistical complexity higher; interpretation careful।

      

    


    

      প্র ০৪"Multi-armed bandit — কখন A/B-এর চেয়ে ভাল?"
      

        
MAB — adaptive allocation; best for specific scenarios।

        
MAB advantages:

        
    
          
  • Less waste — bad variants get less traffic।
    • 
          
  • Faster convergence to best variant।
    • 
          
  • "Always-on" — continuous improvement।
    • 
        

        
MAB disadvantages:

        
    
          
  • Statistical analysis complex (no clean p-value)।
    • 
          
  • Regulator-unfriendly — "what was the experiment exactly"।
    • 
          
  • Implementation harder (bandit algorithm)।
    • 
        

        
When MAB best:

        
    
          
  • Many variants (5+) — A/B sample size huge।
    • 
          
  • Limited traffic — every wasted impression matters।
    • 
          
  • Ad-style content — direct revenue per impression।
    • 
          
  • Continuous personalization (contextual bandit)।
    • 
        

        
When A/B best:

        
    
          
  • 2-3 variants।
    • 
          
  • Long-term metric (delayed reward)।
    • 
          
  • Statistical rigor required।
    • 
          
  • Regulator audit।
    • 
        

        
Hybrid:

        
    
          
  • MAB for exploration + A/B for confirmation।
    • 
          
  • MAB-narrow to top 2 → A/B for final decision।
    • 
        

        
BD example:

        
    
          
  • Daraz banner placement — 10 variants — MAB suit।
    • 
          
  • bKash fraud model selection — A/B (rigor needed)।
    • 
        

        
মূল উপলব্ধি: MAB ≠ replacement for A/B। Different tools, different scenarios। Many variants + immediate reward → MAB; few variants + rigor → A/B।

      

    

  


  

    
 অনুশীলন

    
    
      
  1. Sample size: Conversion 5%, MDE 10% relative — sample size? Daily 10K traffic, kt din?
        
    উপরের code adapt: baseline=0.05, mde=0.10 → ~30,000 per arm। 10K daily / 2 = 5K per arm = 6 din।
    
      
    2. 
      
  2. Pre-register: A/B test plan template — primary metric, MDE, duration, decision criteria।
        
    Hypothesis statement, primary metric definition, guardrail metrics (3+), MDE, sample size, duration, success criteria (e.g., "p < 0.05 AND lift > MDE")।
    
      
    3. 
      
  3. চিন্তা: Pathao surge model A/B — pitfalls কী এড়াবেন?
        
    
          
      
            
    • Network effect — driver-passenger marketplace, cluster by city।
      • 
            
    • Time-of-day variation — full week test।
      • 
            
    • Holiday distortion — exclude special days।
      • 
            
    • Guardrails: complaint rate, driver satisfaction।
      • 
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন

    
  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ২২ · Canary & blue-green