A/B testing — পরিকল্পনা ও বিশ্লেষণ
এই পাঠে যা শিখবেন
- A/B test framework — hypothesis থেকে decision
- Sample size & duration calculation — power analysis
- Result analysis — significance + practical effect
- ৫টি common mistake এবং কীভাবে এড়ানো
১ · A/B test কী এবং কেন
A/B testA/B TestOnline RCT — user-দের randomly দু'টি (বা বেশি) variant-এ ভাগ। Tech industry-তে product decision-এর gold standard। Booking.com, Netflix, Daraz প্রতিদিন শতশত A/B test চালায়। = digital RCT। User random-ভাবে দু'টি variant-এ ভাগ — control (existing) vs treatment (new feature)। Outcome compare।
কেন গুরুত্বপূর্ণ:
- Causal evidence — observational analysis-এর confounder নেই।
- Quick — software change → days-এ result।
- Scalable — millions of user evaluate possible।
- Bottom-line driven — business metric সরাসরি measure।
২ · Pre-test design
Step ১ — Hypothesis:
- "Daraz home page-এ recommended product carousel দেখালে — purchase rate বাড়বে।"
- Specific, testable, directional।
Step ২ — Primary metric:
- One metric — decision-এর ভিত্তি।
- Daraz: purchase conversion। Pathao: ride-completion। bKash: transaction success।
Step ৩ — Guardrail metric:
- Side-effect monitor — যেমন: page-load time, app crash, complaint rate।
- Primary improve হলেও guardrail degrade হলে — launch করা যাবে না।
Step ৪ — MDE (Minimum Detectable Effect):
- "Smallest effect — যা business-meaningful এবং আমরা detect করতে চাই।"
- Daraz-এ ০.৫% absolute lift may be MDE।
- Smaller MDE → larger sample size।
৩ · Sample size calculation
Two-proportion test-এর সাধারণ formula:
$$n_\text{per arm} = \frac{2 \cdot \bar{p}(1-\bar{p}) \cdot (z_{\alpha/2} + z_\beta)^2}{\text{MDE}^2}$$
যেখানে:
- $\bar{p}$ = baseline conversion rate।
- $z_{\alpha/2}$ = ১.৯৬ (৯৫% confidence)।
- $z_\beta$ = ০.৮৪ (৮০% power)।
উদাহরণ: baseline conversion ৫%, MDE ০.৫% absolute (১০% relative lift), α=০.০৫, power=০.৮:
$$n \approx \frac{2 \times 0.05 \times 0.95 \times (1.96+0.84)^2}{(0.005)^2} \approx 29{,}800 \text{ per arm}$$
মোট ~৬০,০০০ user দরকার!
৪ · Power analysis
Statistical powerStatistical Power$1 - \beta$ = "true effect থাকলে correctly detect করার probability"। Industry standard ৮০%। Underpowered test inconclusive — sample size key। = "যদি true effect থাকে — correctly detect করার probability"। Standard: ৮০%।
Power বাড়াতে:
- Sample size বাড়ান।
- MDE বড় করুন (less ambitious)।
- Outcome variance কমান (CUPED, stratified analysis)।
- α relax করুন (০.১) — সাবধান।
৫ · Test duration
Sample size-এর সাথে duration-ও বাছতে হয়:
- Daily traffic ১০,০০০ → ৬০,০০০ sample = ৬ দিন।
- সাবধান: minimum ৭ দিন (full week effect)।
- Holiday/event-এ run এড়ান।
- Multi-week — novelty wear off, long-term effect।
৬ · Result analysis
Decision matrix:
- p < ০.০৫ + lift ≥ MDE + guardrail OK → launch।
- p < ০.০৫ + lift < MDE → significant but trivial → consider।
- p ≥ ০.০৫ → inconclusive। আরও test বা skip।
- Guardrail violation → no launch regardless।
Report: point estimate, ৯৫% CI, p-value, effect size, sample size, duration।
৭ · ৫টি common pitfall
- (১) Peeking: Test চলাকালীন বারবার p-value check। Significant হওয়ামাত্র stop। False positive rate ৫% → ২০%+ ঝুঁকি। Sequential test বা pre-fixed duration ব্যবহার।
- (২) Multiple metrics: ১০ metric check, একটিতে significant → "winner"। Bonferroni correction।
- (৩) Novelty effect: New feature-এ initial spike, পরে wear off। Multi-week test।
- (৪) Network interference: Pathao driver-এর A treatment অন্য driver-এর B-কে affect। Cluster randomization।
- (৫) Segment fishing: Overall null, কিন্তু "Sylhet female 25-34"-এ significant → exploratory only।
৮ · বাংলাদেশী case studies
bKash:
- QR-code payment flow redesign। Primary: completion rate। Guardrail: error rate। Test: ৪ সপ্তাহ, ৫ লাখ users।
- "Send Money"-এর confirmation step add — fraud কমানোর জন্য। Friction increase কিন্তু safety up।
Daraz:
- "Add to cart" button-এর color test — orange vs red।
- Search result personalized ranking।
- Discount strategy — flat vs tiered।
Pathao:
- Driver dispatch algorithm — distance vs ETA।
- Pricing — surge level।
- Notification frequency — fatigue threshold।
৯ · Sample size — Python-এ
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize
# Daraz baseline 5% conversion, target detect 0.5% absolute lift
p_control = 0.05
p_treatment = 0.055
mde = p_treatment - p_control
# Effect size (Cohen's h for proportions)
h = proportion_effectsize(p_treatment, p_control)
# Power analysis
analysis = NormalIndPower()
n = analysis.solve_power(effect_size=h, alpha=0.05, power=0.8,
alternative="two-sided")
print(f"Baseline : {p_control*100:.1f}%")
print(f"Target : {p_treatment*100:.1f}%")
print(f"Absolute MDE : {mde*100:.2f}%")
print(f"Effect size (h): {h:.4f}")
print(f"Sample per arm : {int(n):,}")
print(f"Total : {int(2*n):,}")
print(f"\n@ 10,000 daily users → {int(2*n)/10000:.1f} days")১০ · Result analysis — Python-এ
import numpy as np
from scipy import stats
from statsmodels.stats.proportion import proportions_ztest, proportion_confint
# Daraz A/B test result
n_control = 30_000
conv_control = 1_500 # 5.00%
n_treatment = 30_000
conv_treatment = 1_710 # 5.70%
# z-test for proportions
counts = np.array([conv_treatment, conv_control])
nobs = np.array([n_treatment, n_control])
z_stat, p_val = proportions_ztest(counts, nobs)
# CI on difference
p_t = conv_treatment / n_treatment
p_c = conv_control / n_control
diff = p_t - p_c
se_diff = np.sqrt(p_t*(1-p_t)/n_treatment + p_c*(1-p_c)/n_control)
ci_low = diff - 1.96 * se_diff
ci_high = diff + 1.96 * se_diff
print(f"Control : {p_c*100:.2f}% ({conv_control}/{n_control})")
print(f"Treatment : {p_t*100:.2f}% ({conv_treatment}/{n_treatment})")
print(f"Lift : +{diff*100:.2f}pp ({diff/p_c*100:+.1f}% relative)")
print(f"95% CI : ({ci_low*100:+.2f}, {ci_high*100:+.2f}) pp")
print(f"z-stat : {z_stat:.3f}")
print(f"p-value : {p_val:.4f}")
if p_val < 0.05:
print("\n→ Significant. Lift positive. Recommend launch (after guardrail check).")
else:
print("\n→ Not significant. Inconclusive.")ভাবনার প্রশ্ন
প্রতিটি প্রশ্ন নিজে কিছুক্ষণ ভাবুন — তারপর "→ উত্তর" চাপুন।
প্র ০১ "Peeking"-এর সমস্যা কী এবং কেন এত বিপজ্জনক? Sequential testing-এর মাধ্যমে কীভাবে solve হয়?
Peeking — A/B testing-এর সবচেয়ে subtle এবং widespread mistake। এমনকি সিনিয়র data scientist-রাও করেন।
সমস্যা:
- Test চলাকালীন বারবার p-value check।
- "Significant হলো — stop! Launch!" — তাড়াহুড়ো decision।
- Each peek = additional test → multiple comparisons।
- Nominal α = ০.০৫ → actual false positive rate ২০-৫০%!
কেন এত high false positive:
- Random walk-এ p-value over time fluctuate।
- Eventually most random walks "barely" cross ০.০৫।
- Continuous monitoring → guaranteed cross at some point।
- Mathematically — α explode।
Concrete simulation:
- True effect = ০। Two-arm test।
- 1 peek at end: ৫% reject H0 (correct)।
- 10 peeks during test: ~১৯% reject।
- 50 peeks: ~৩৩% reject।
- Continuous: → ১০০% eventually।
Solutions:
- (১) Pre-fixed sample size: Most rigid। Calculate sample, run until reached, look once। Industry default।
- (২) Bonferroni correction: If you must peek k times, use α/k each। Simple but conservative।
- (৩) Group sequential design: Pocock or O'Brien-Fleming boundaries। Pre-determined "look points" with adjusted thresholds।
- (৪) Alpha spending function: Continuous monitoring with controlled total α। Lan-DeMets approach।
- (৫) Bayesian: Posterior probability — peeking-resistant framework। Optional stopping OK।
- (৬) mSPRT (mixture Sequential Probability Ratio Test): Optimized for tech industry। Used by Optimizely, Microsoft।
Industry practice:
- Booking.com — pre-fixed duration, no peeking, full report at end।
- Microsoft ExP — Bayesian-ish + early-stopping rules।
- Optimizely — built-in always-valid p-values (mSPRT)।
- Daraz/Pathao — varies; less mature, peeking common (mistake)।
"Always-valid" inference:
- Modern technique — sequential test যা যেকোনো sample-size-এ valid।
- Confidence sequence — narrows over time।
- Howard, Ramdas, McAuliffe (২০২১) papers — mathematical foundations।
- Practitioner-friendly: Splitio, Eppo, Statsig — built-in।
Practical guidance:
- Junior team: pre-fixed duration, no peeking।
- Mature team: sequential design tools (always-valid p-values)।
- Organization buy-in: leadership accept "wait for full sample"।
মূল উপলব্ধি: Peeking — invisible alpha-inflation। Without correction, "data-driven culture" = "false positive culture"।
প্র ০২ Pathao-এর driver-side A/B test-এ — driver A-এর treatment driver B-কে কীভাবে affect করে? Network interference solve করতে কী করবেন?
Network interference — two-sided marketplace A/B testing-এর গুরুতর সমস্যা।
Pathao-specific মেকানিজম:
- Resource competition: Limited rider pool। Driver A treatment-এ বেশি accept → fewer rides for driver B।
- Pricing spillover: Treatment driver discount-এ আকর্ষণীয় → control driver-এর demand কমে।
- Reputation effect: Treatment-এর better service → overall Pathao perception improve → benefits all।
- Geographic clustering: Same area-এ A ও B driver — they compete।
সমস্যা:
- SUTVA assumption (Stable Unit Treatment Value Assumption) violated।
- Treatment effect overestimated বা underestimated।
- Naive A/B → wrong launch decision।
Detection:
- "Switchback test": same treatment-এ different times।
- Effect varies with treatment-share।
- If 50/50 split-এ effect ভিন্ন from 90/10 — interference।
Solutions:
-
(১) Cluster randomization: Geographic area, time-bucket, social network — entire cluster A or B। Spillover within cluster contained।
- Pathao: ঢাকা area-A vs area-B। Each area homogeneous treatment।
- Cost: variance বাড়ে, sample-size requirement বাড়ে।
-
(২) Switchback test: Same area, time-windows alternate — week 1 A, week 2 B, week 3 A...
- Same population, no contamination।
- Drawback: temporal trend confound।
- (৩) Time-of-day randomization: Morning A, afternoon B।
- (৪) Synthetic control: Treated area-এর "synthetic" comparison — অন্য area-গুলোর weighted average।
- (৫) Network-aware analysis: Estimate spillover explicitly। Aronow & Samii (২০১৭) framework।
Pathao concrete recommendation:
- Driver-pricing test: switchback by week (city-wide)।
- Dispatch algorithm: cluster by area।
- Rider-side feature: standard A/B (less interference)।
- Notification frequency: switchback by day-of-week।
Lyft, Uber-এর experience:
- Bojinov et al. (২০২৩, Lyft): switchback testing standard।
- Uber's Causal Forest — heterogeneous treatment estimate।
- DoorDash — "marketplace experiments" team dedicated।
Modern best practice:
- Specialized tools — Lyft's "switchback experiments" framework।
- Causal inference + simulation।
- Always quantify interference magnitude before scaling।
মূল উপলব্ধি: Marketplace business-এ naive A/B prone to interference। Cluster + switchback + network analysis — multi-pronged approach।
প্র ০৩ "Novelty effect"-এর কারণে নতুন feature প্রথম সপ্তাহে winning দেখা যায়, পরে drop। কীভাবে চিনবেন এবং long-term effect measure করবেন?
Novelty effect — A/B testing-এর সবচেয়ে subtle আকর্ষণ-পতন।
Mechanism:
- New feature curiosity-attract করে।
- User explore — temporary engagement spike।
- Wear off — habit formation না হলে engagement decline।
- Long-run effect প্রায়ই neutral বা negative।
Daraz example:
- Home page-এ new "AI recommendation" carousel।
- Week 1: click-through rate ১৫% (vs control ৮%) — winning!
- Week 2: ১২%। Week 3: ৯%। Week 4: ৮.৫% — almost equal।
- Decision: launch করলে initial bump expected, sustained gain doubtful।
Detection:
- (১) Time-trend plot: Daily metric of treatment-control। Slope visible।
- (২) New vs returning user split: New users novelty-prone; returning user-এ সম্ভবত actual effect।
- (৩) Long test duration: ৪+ weeks; প্রথম 2 weeks ignore।
- (৪) Holdout period: Launch করার পর ৫% user permanently control — long-term measure।
Long-term effect-এর challenge:
- Months-long test impractical — many feature compete।
- Holdout group business cost (control না পেলে opportunity loss)।
- Long-run metric (retention, lifetime value) noisy।
Solutions:
- Pre-specified analysis: Week 4-6-এর effect primary। Pre-register।
- Multiple horizons: Day 1, Week 1, Month 1 — all report। Trend explicit।
- Holdback experiments: Post-launch hold-out group। Apple-, Facebook-এ standard।
- Surrogate metrics: Long-run outcome predict-করা early metrics। Athey et al. (2019)।
Opposite — "primacy effect":
- New feature initially confuse — engagement drop।
- Learning curve পরে recover ও exceed।
- Same long-run analysis solve।
Bangladesh-specific examples:
- bKash QR-payment launch — novelty 2 weeks, then habit formation;.
- Pathao food delivery launch — initial frenzy, then plateau।
- Daraz dark-mode — novelty, but accessibility-genuine winners stay।
Counterpoint — when novelty IS the goal:
- Promotional campaign — short-term spike fine।
- One-time event (Black Friday) — novelty acceptable।
- Acquisition push — initial response key।
মূল উপলব্ধি: Short-term winners ≠ long-term winners। Pre-register multi-horizon analysis; holdback for sustained measurement।
প্র ০৪ "Test result negative — feature kill" — সবসময় কি সঠিক? Negative result-এর কী insight থাকতে পারে এবং business-এ কীভাবে use করবেন?
Negative result — undervalued asset। Most companies waste এই knowledge।
Negative result-এর information:
- (১) "Feature does not work": Hypothesis disproved। Direct knowledge।
- (২) "Feature works only in specific segment": Heterogeneous effect — sub-group-এ launch।
- (৩) "Feature works but with cost": Primary up, guardrail violated — refine।
- (৪) "Inconclusive — underpowered": Bigger test or different design।
Decision matrix:
- p > ০.০৫ + small effect + tight CI → likely no effect → kill।
- p > ০.০৫ + moderate effect + wide CI → underpowered → expand test।
- p < ০.০৫ + negative effect + guardrail OK → kill, but learn।
- Heterogeneous effect (some segment +, some −) → segmented launch।
Daraz scenario:
"AI-recommendation carousel" — overall null। But:
- Power user (10+ orders/month): +১০% engagement।
- New user: −৫% (overwhelmed, confused)।
- Lesson: launch only for power user; redesign for new users।
Negative result-এর hidden value:
- Mental model update: "We thought users want X — they don't"।
- Avoid duplicate work: Other team-এর similar idea pre-empt।
- Strategic insight: Theory underlying এর correctness check।
- Investment redirection: Resources elsewhere।
- Hypothesis refinement: "Why didn't it work?" — diagnostic।
Why companies underuse:
- Negative result published কম — bias।
- "My feature failed" — career risk।
- No structured way to capture learning।
- Org culture — speed obsession।
Best-practice — "experiment journal":
- প্রতিটি test result + interpretation + next-step documented।
- Searchable knowledge base।
- Quarterly review — pattern identify।
Industry examples:
- Microsoft ExP — internal "Negative results" library।
- Booking.com — "Most experiments fail" culture; data-driven humility।
- Spotify — "Insights repository" with all results।
Diagnostic for negative result:
- Implementation check: Treatment actually delivered? Bug?
- Sample-ratio mismatch: A/B split actually 50/50? — bias if not।
- Outcome measurement: Right metric?
- Heterogeneity: Sub-group-এ effect?
- Power: Underpowered?
Bangladesh-specific:
- Pathao surge pricing — failed in tier-2 city, success in Dhaka।
- Daraz Bengali UI — failed for English-comfort user, success for new user।
- bKash crypto-feature — universally negative; killed; saved investment।
Communication template:
- "Hypothesis X tested. Result: −২% lift, p = ০.০৩, CI [−৩%, −১%]।"
- "Guardrail OK. Decision: do not launch।"
- "Learning: assumption Y likely wrong; future: explore Z।"
মূল উপলব্ধি: Negative result = data, not failure। Systematic learning capture = competitive advantage।
অনুশীলন
-
Sample size: bKash baseline conversion ৩%। আপনি ০.৩% absolute lift detect করতে চান। ৯৫% confidence, ৮০% power। Sample per arm কত?
from statsmodels.stats.proportion import proportion_effectsize from statsmodels.stats.power import NormalIndPower h = proportion_effectsize(0.033, 0.030) n = NormalIndPower().solve_power(effect_size=h, alpha=0.05, power=0.8) print(int(n)) # ~50,000-55,000 per armমোট ~১ লাখ user। ১০,০০০ daily traffic-এ ১০ দিন।
-
Result interpret: Pathao test: control 88%, treatment 89%, n = ৫,০০০ per arm, p = ০.১৫।
p > ০.০৫ — significant না। Lift ১pp (relative ~১.১%)। Sample possibly underpowered। Options: (১) extend test sample, (২) declare null and kill, (৩) check heterogeneous effect by segment। Effect direction positive — promising লক্ষণ but not proof।
-
Design: Daraz "checkout button color" test design করুন: hypothesis, primary metric, MDE, guardrail, sample size, duration, success criteria।
- Hypothesis: Orange-red CTA → checkout completion ↑।
- Primary: Checkout completion rate।
- Guardrail: page-load time, error rate, refund rate।
- MDE: ০.৫% absolute lift।
- Sample: baseline ১০% → ~৩০,০০০ per arm = ৬০,০০০ total।
- Duration: ৭-১০ days (full week + buffer)।
- Success: p < ০.০৫ + lift ≥ MDE + guardrail OK → launch।
আরও পড়ুন · ABCL TECH-এ আপনার পরবর্তী পদক্ষেপ
- পাঠ ১৬ · Exploratory Data Analysis পরবর্তী পাঠ পরবর্তী মডিউল — feature engineering ও EDA।
- পাঠ ১৪ · Correlation ও causation আগের পাঠ A/B test = causation establish-এর gold standard।
- পাঠ ১২ · t-test এই পাঠের সাথে সম্পর্কিত A/B test result analyze — t-test বা proportion test।
- সব AI Courses দেখুন ABCL TECH Python, ML, DL, NLP, CV, GenAI, RL, MLOps — সব AI কোর্স একসাথে।