পাঠ ২৩ · ৩৩-এর মধ্যে · মডিউল ৩
Home / AI Courses / MLOps / A/B testing

A/B testing for ML

A/B testing — hypothesis to decision
৭ মিনিট পড়া উচ্চ · Advanced Stats

এই পাঠে যা শিখবেন

  • Hypothesis + MDE + sample size calculation
  • Primary vs guardrail metrics
  • Pitfalls — peeking, novelty, attribution
  • Multi-armed bandit alternative

১ · A/B vs Canary

Canary = deployment safety mechanism (does new version break?)। A/B test = product/business decision (does new version improve metrics?)।

  • Canary: short, infrastructure metric (latency, error).
  • A/B: long (days/weeks), business metric (conversion, revenue).
  • Canary first, then A/B-এ promote.

২ · Hypothesis design

একটি good hypothesis:

  • Null (H0): new model = control. No improvement.
  • Alternate (H1): new model conversion +5%.
  • Primary metric: conversion rate.
  • Guardrails: latency < baseline+20ms; error rate < baseline.
  • Population: all users in BD region.
  • Duration: 14 days (capture weekly cycle).

৩ · MDE ও sample size

MDEMinimum Detectable Effectsmallest improvement test can statistically detect with given power. Smaller MDE → bigger sample size. = "smallest improvement we care about".

$$ n \approx \frac{2 \sigma^2 (z_{\alpha/2} + z_\beta)^2}{\text{MDE}^2} $$

Per arm sample size. For binary metric (conversion):

$$ n \approx \frac{2 p (1-p) (z_{\alpha/2} + z_\beta)^2}{\text{MDE}^2} $$

৪ · Sample size calculation

Python · Sample size 
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

# Current conversion 8%
# Want to detect 5% relative improvement → 8% → 8.4%
baseline = 0.08
mde_relative = 0.05
treatment = baseline * (1 + mde_relative)

effect = proportion_effectsize(baseline, treatment)
analysis = NormalIndPower()
n_per_arm = analysis.solve_power(
    effect_size=effect,
    alpha=0.05,
    power=0.80,
    alternative="two-sided",
)
print(f"Per arm sample size: {int(n_per_arm):,}")

# At 1000 users/day per arm = days_needed
total_traffic = 50_000  # daily users
split = 0.5  # 50/50
daily_per_arm = total_traffic * split
days = n_per_arm / daily_per_arm
print(f"Duration: ~{days:.1f} days")

    


    

      Realistic numbers: 8% conversion + 5% relative MDE → ~80,000 per arm needed. At 25,000/day → ~3 days. Power 80%, α 5% standard.
    


    
৫ · Primary vs guardrail


    
    
      
  • Primary: the metric we want to improve.
    • 
      
  • Guardrails: shouldn't regress.
    • 
    


    

      Recommendation example:
    

    
    
      
  • Primary: click-through rate.
    • 
      
  • Guardrails: page load latency, bounce rate, complaint rate.
    • 
    


    
৬ · Common pitfalls


    
    
      
  • Peeking: daily check + early stop on win = false positive. Pre-register decision time.
    • 
      
  • Sample ratio mismatch (SRM): 50/50 expected, 49/51 observed — assignment bug? Check chi-square.
    • 
      
  • Novelty effect: new feature attention boost initial; fades. Run longer.
    • 
      
  • Carry-over: user-level randomization দরকার, not session.
    • 
      
  • Network effects: social product — control + treatment user-influence each other.
    • 
    


    
৭ · Multi-armed bandit alternative


    

      MAB = adaptive allocation. Better-performing arm → more traffic. Reduce regret.
    

    
    
      
  • Pros: faster convergence, less wasted traffic on bad variants.
    • 
      
  • Cons: harder analysis, statistical significance complex.
    • 
      
  • Use when: many variants, traffic limited, "maximize during test".
    • 
      
  • Skip when: classical A/B preference (regulator-friendly).
    • 
    


    
৮ · ML metric vs business metric


    

      Often divergent.
    

    
    
      
  • New model AUC +2% (offline) → CTR +0% (online).
    • 
      
  • Reasons: offline test ≠ online distribution; user behavior change.
    • 
      
  • Always trust online business metric over offline ML metric.
    • 
    


    

      
        
        A/B test workflow — design → run → decide

        
        Design
        hypothesis, MDE

        
        Sample size
        power calc

        
        Randomize
        user-level hash

        
        Run experiment
        2 weeks

        
        SRM check
        assignment OK?

        
        Compute metrics
        primary + guardrail

        
        Stat test
        CI, p-value

        
        Decision
        launch / hold / kill

        
        
        
        
        
        
        

        Pre-register decision criteria avoid post-hoc rationalization.
      
      
A/B test workflow — design statistically up-front; run end-to-end; decision criteria pre-registered.

    


    

      Bangladesh-এ A/B test platform — Daraz, Pathao, bKash, Foodpanda — সবাই custom বা Optimizely-style external use.
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      প্র ০১"ML metric +5% but business metric flat — কেন? কী করবেন?"
      

        
Common ML productivity paradox.

        
Why divergence:

        
    
          
  • Offline test set distribution ≠ online production distribution.
    • 
          
  • Selection bias — recommendation only logged for shown items.
    • 
          
  • User behavior adaptation — better recommendation, but user not click more.
    • 
          
  • Confounding — recommendation only one of many factors.
    • 
          
  • Saturation — user already at maximum daily clicks.
    • 
        

        
Investigation steps:

        
    
          
  • Subgroup analysis — maybe wins concentrate in some segments.
    • 
          
  • Time-of-day analysis — peak vs off-peak ভিন্ন।
    • 
          
  • Funnel — click increased but conversion not?
    • 
          
  • Counterfactual — "if old model would have shown different items".
    • 
        

        
Action options:

        
    
          
  • Hold — A/B test data not strong enough; extend duration.
    • 
          
  • Ship anyway — ML metric improvement long-term value.
    • 
          
  • Iterate — discovered offline-online gap; fix offline metric.
    • 
          
  • Kill — ROI not justifying complexity.
    • 
        

        
BD context — Daraz example:

        
    
          
  • NDCG +5% offline → CTR flat online.
    • 
          
  • Investigation: Daraz users prefer cheap items; new model more diverse but pricier.
    • 
          
  • Fix: business-aware loss function — price × CTR optimize.
    • 
        

        
মূল উপলব্ধি: Offline-online gap normal. Trust online business metric. Use offline for sanity-check, online for ship decision.

      

    


    

      প্র ০২"Long-term effects — A/B test 2 weeks, but real impact 6 months পরে?"
      

        
Long-term effect কঠিন measure কিন্তু critical.

        
Examples:

        
    
          
  • Aggressive engagement model → short-term DAU up → long-term burnout.
    • 
          
  • Push notification volume → click-through up → unsubscribe later.
    • 
          
  • Aggressive ad placement → revenue up → user trust down.
    • 
        

        
Strategies:

        
(১) Holdout group long-term:

        
    
          
  • 5% users never see treatment, even after launch.
    • 
          
  • 6 month after — compare retention, lifetime value.
    • 
          
  • Cost: small group continuously held back.
    • 
        

        
(২) Surrogate metrics:

        
    
          
  • Short-term proxies for long-term.
    • 
          
  • e.g., "user who clicks unsubscribe in week 1 = lost user month 6".
    • 
          
  • Validate proxy initially with retrospective.
    • 
        

        
(৩) Phased rollout with monitoring:

        
    
          
  • Launch but continue monitoring 3-6 months.
    • 
          
  • Long-term metric degrade → rollback.
    • 
        

        
(৪) Switchback experiments:

        
    
          
  • Toggle on/off weekly — measure lift each cycle.
    • 
          
  • Long-term carryover insight.
    • 
        

        
BD context:

        
    
          
  • Bangladesh subscription business — retention 6-12 month most important.
    • 
          
  • Holdout group + surrogate combination.
    • 
        

        
মূল উপলব্ধি: Long-term measurement structural commitment. Holdout permanent, surrogate validated, post-launch monitoring continued. Skip — eventually quality degrades silently.

      

    


    

      প্র ০৩"Network effects — social product-এ A/B testing চ্যালেঞ্জ।"
      

        
Social/network products — control + treatment interact. Standard A/B violates assumption.

        
Examples:

        
    
          
  • "Better feed algorithm" → user post more → control's feed change-ও।
    • 
          
  • Marketplace — better seller experience → buyers see better products → control buyers benefit.
    • 
        

        
Solutions:

        
(১) Cluster randomization:

        
    
          
  • Assign by geography (city, region).
    • 
          
  • Within cluster, single condition — interactions stay within.
    • 
          
  • Trade-off: bigger sample needed; fewer "users" for statistical purposes.
    • 
        

        
(২) Time-based switchback:

        
    
          
  • Whole platform alternates control/treatment per hour or day.
    • 
          
  • Used by Uber, Lyft (matching algorithm).
    • 
          
  • Requires no carryover-effects.
    • 
        

        
(৩) Ego-network analysis:

        
    
          
  • Friends-of-friends control.
    • 
          
  • Account treatment effect on neighbors.
    • 
        

        
BD context:

        
    
          
  • Pathao — driver-passenger marketplace; cluster by city.
    • 
          
  • Daraz — seller-buyer; product category cluster.
    • 
        

        
মূল উপলব্ধি: Network products — standard A/B mismeasures. Cluster or switchback essential. Statistical complexity higher; interpretation careful.

      

    


    

      প্র ০৪"Multi-armed bandit — কখন A/B-এর চেয়ে ভাল?"
      

        
MAB — adaptive allocation; best for specific scenarios.

        
MAB advantages:

        
    
          
  • Less waste — bad variants get less traffic.
    • 
          
  • Faster convergence to best variant.
    • 
          
  • "Always-on" — continuous improvement.
    • 
        

        
MAB disadvantages:

        
    
          
  • Statistical analysis complex (no clean p-value).
    • 
          
  • Regulator-unfriendly — "what was the experiment exactly".
    • 
          
  • Implementation harder (bandit algorithm).
    • 
        

        
When MAB best:

        
    
          
  • Many variants (5+) — A/B sample size huge.
    • 
          
  • Limited traffic — every wasted impression matters.
    • 
          
  • Ad-style content — direct revenue per impression.
    • 
          
  • Continuous personalization (contextual bandit).
    • 
        

        
When A/B best:

        
    
          
  • 2-3 variants.
    • 
          
  • Long-term metric (delayed reward).
    • 
          
  • Statistical rigor required.
    • 
          
  • Regulator audit.
    • 
        

        
Hybrid:

        
    
          
  • MAB for exploration + A/B for confirmation.
    • 
          
  • MAB-narrow to top 2 → A/B for final decision.
    • 
        

        
BD example:

        
    
          
  • Daraz banner placement — 10 variants — MAB suit.
    • 
          
  • bKash fraud model selection — A/B (rigor needed).
    • 
        

        
মূল উপলব্ধি: MAB ≠ replacement for A/B. Different tools, different scenarios. Many variants + immediate reward → MAB; few variants + rigor → A/B.

      

    

  


  

    
 অনুশীলন

    
    
      
  1. Sample size: Conversion 5%, MDE 10% relative — sample size? Daily 10K traffic, kt din?
        
    উপরের code adapt: baseline=0.05, mde=0.10 → ~30,000 per arm. 10K daily / 2 = 5K per arm = 6 din.
    
      
    2. 
      
  2. Pre-register: A/B test plan template — primary metric, MDE, duration, decision criteria.
        
    Hypothesis statement, primary metric definition, guardrail metrics (3+), MDE, sample size, duration, success criteria (e.g., "p < 0.05 AND lift > MDE").
    
      
    3. 
      
  3. চিন্তা: Pathao surge model A/B — pitfalls কী এড়াবেন?
        
    
          
      
            
    • Network effect — driver-passenger marketplace, cluster by city.
      • 
            
    • Time-of-day variation — full week test.
      • 
            
    • Holiday distortion — exclude special days.
      • 
            
    • Guardrails: complaint rate, driver satisfaction.
      • 
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন

    
  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ২২ · Canary & blue-green