ML-এর জন্য পরিসংখ্যান — অনুমান ও পরীক্ষা

এটি statistics-এর সবচেয়ে interesting historical debate. ১৯২০-৫০-এ Fisher, Neyman, Pearson — frequentist; Jeffreys, Bayes-এর shadow — Bayesian. ML এই বিভাজনকে partly transcend করেছে।

(১) Frequentist দর্শন:

• Probability = long-run frequency.
• Parameter $\theta$ — fixed but unknown.
• Data — random.
• Hypothesis test, confidence interval, p-value — frequentist tools.
• "Given infinite data, how often I'd be right?"

(২) Bayesian দর্শন:

• Probability = degree of belief.
• Parameter $\theta$ — random with distribution.
• Prior + likelihood = posterior (Bayes' theorem)।
• "Given my data, what's my belief about parameters?"

(৩) ML-এ কোনটা?

• Surprisingly mixed:
• Cross-entropy loss = MLE → frequentist root.
• L2 regularization = Gaussian prior → Bayesian root.
• Dropout = approximate Bayesian inference.
• VAE, BNN — explicitly Bayesian.
• Deep learning practitioners use both, often unknowingly.

(৪) কখন কোনটা practical?

• Frequentist (large data): A/B testing, classical ML — sample size বড় হলে dominant.
• Bayesian (small data, prior knowledge): Drug trials, expert systems.
• Bayesian (uncertainty quantification): Active learning, safety-critical AI.

(৫) আজকের bridge — Empirical Bayes:

• Prior data থেকেই estimate.
• Hierarchical models — Bayesian framework, frequentist parameter estimation.
• Stein's paradox — অনেক parameter একসাথে estimate করলে — Bayesian shrinkage ভাল।

(৬) Deep Learning-এ:

• Standard NN: Frequentist (point estimate)।
• BNN (Bayesian NN): Weight-এর posterior.
• MC Dropout (Gal & Ghahramani ২০১৬): Dropout = Bayesian inference.
• Deep Ensembles: Approximate Bayesian, easier.
• Conformal Prediction: Frequentist uncertainty quantification.

(৭) Real-world implications:

• Medical AI — Bayesian preferred (uncertainty critical)।
• Self-driving cars — Bayesian uncertainty for safety.
• Recommendation — frequentist (huge data)।
• RLHF — Bayesian-ish KL constraints.

(৮) Bangladesh context:

• Small data scenarios — Bayesian critical.
• Bangla NLP-এ pre-trained model + small fine-tune = Bayesian flavor.
• Medical diagnosis app — Bayesian uncertainty mandatory.

মূল উপলব্ধি: Frequentist vs Bayesian — অর্ধশতক ধরে statistics-এর "religion war"। আজকের ML practical — দু'টিই ব্যবহার, প্রায় unconsciously. ভাল ML engineer দু'টির philosophical foundation বুঝে — সঠিক problem-এ সঠিক approach.

এই connection ML-এর সবচেয়ে elegant theoretical insights-এর একটি। Linear regression থেকে deep learning পর্যন্ত প্রায় সব অংশে এর প্রতিফলন।

(১) Setup:

• Linear regression: $y = w^T x + \epsilon$, $\epsilon \sim N(0, \sigma^2)$।
• Weights $w$ on prior: $w \sim N(0, \tau^2 I)$ (Gaussian, mean 0)।
• Goal: posterior $P(w | D) = \dfrac{P(D|w) P(w)}{P(D)}$।

(২) Likelihood:

$P(D|w) = \prod_i N(y_i | w^T x_i, \sigma^2) \propto \exp\left(-\dfrac{1}{2\sigma^2}\sum_i (y_i - w^T x_i)^2\right)$

(৩) Prior:

$P(w) = N(0, \tau^2 I) \propto \exp\left(-\dfrac{1}{2\tau^2} \|w\|^2\right)$

(৪) MAP — log নিয়ে:

$\log P(w|D) = -\dfrac{1}{2\sigma^2}\sum_i (y_i - w^T x_i)^2 - \dfrac{1}{2\tau^2}\|w\|^2 + \text{const}$

$\arg\max$ = $\arg\min$ of negative:

$\hat{w} = \arg\min_w \left[\sum_i (y_i - w^T x_i)^2 + \dfrac{\sigma^2}{\tau^2}\|w\|^2\right]$

সরাসরি ridge regression! $\lambda = \sigma^2/\tau^2$ — regularization strength.

(৫) তাৎপর্য:

• $\tau$ ছোট = strong prior (weights small) = $\lambda$ বড় (more regularization)।
• $\tau$ বড় = weak prior = $\lambda$ ছোট = closer to MLE.
• $\tau \to \infty$: pure MLE.

(৬) L1 regularization?

• L1 = Laplace prior: $P(w) \propto \exp(-|w|/b)$।
• Sharp peak at 0 → sparse weights.
• "Most weights are exactly zero" prior.

(৭) Practical implications:

• Hyperparameter বুঝা: $\lambda$ — শুধু "magic number" না, prior-এর strength.
• Bayesian interpretation: Cross-validation $\lambda$ tuning = empirical Bayes.
• Calibrated regularization: Domain knowledge থেকে $\tau$ estimate.
• Group LASSO, Elastic Net: বিভিন্ন priors.

(৮) Deep learning-এ:

• Weight decay (L2) = Gaussian prior on every weight.
• Dropout = stochastic regularization, approximate Bayesian.
• Batch norm = implicit regularization.
• Early stopping = posterior-এর "neighborhood" থেকে stop.

(৯) Beyond Gaussian — Bayesian Deep Learning:

• Variational Inference: explicit posterior approximation.
• MC Dropout: dropout = posterior sampling.
• SWAG: SGD trajectory থেকে posterior approximate.
• Deep Ensembles: multiple models = posterior samples.

(১০) Bangladesh AI context:

• Limited data (Bangla datasets ছোট) — strong prior valuable.
• Pre-trained model fine-tuning — pretrained weights = prior.
• Domain expertise (medical, legal Bangla) — explicit prior helps.

মূল উপলব্ধি: "L2 regularization = Gaussian prior" — শুধু trick না, statistics-এর deep insight. এই connection বুঝলে — hyperparameter tuning informed, regularization interpretable, model uncertainty meaningful. Mature ML engineer-এর mathematical literacy এখানে।

AI evaluation-এ p-value misuse epidemic. Tech blog-এ "model improved by 2%" claims-এর পেছনে প্রায়ই কোনো statistical test নেই। সঠিক approach critical.

(১) Naive comparison সমস্যা:

• "৮৭% vs ৮৫% — তাই ভাল।" — কিন্তু variance?
• Single test set-এর performance noise হতে পারে।
• Different random seeds = different results.
• Cherry-picking risk.

(২) সঠিক methodology:

1. Multiple runs: দু'টি model 10-50 বার train, ভিন্ন seeds.
2. Same test set: Paired comparison.
3. Paired t-test: Per-seed difference Gaussian হলে।
4. Wilcoxon signed-rank test: Non-parametric, robust.
5. Bootstrap: Distribution-free CI.
6. McNemar's test: Per-sample classification difference.

(৩) Python example:

from scipy.stats import ttest_rel, wilcoxon

baseline = [0.847, 0.851, 0.849, 0.852, 0.850, ...]  # 10 runs
new_model = [0.866, 0.871, 0.869, 0.872, 0.868, ...]  # 10 runs

# Paired t-test
t, p = ttest_rel(new_model, baseline)
print(f"t={t:.3f}, p={p:.4f}")
# p < 0.05 হলে "significant" claim করা যায়

(৪) Common ভুল:

• "p-value = probability that $H_0$ is true":
  • ভুল! p-value = $P(\text{data} | H_0)$, not $P(H_0 | \text{data})$।
• "p > 0.05 = no difference":
  • ভুল! Power-এর সমস্যা — small sample-এ significant difference miss.
• "p-hacking":
  • একাধিক test, multiple comparisons — Bonferroni correction.
  • Until significant — fishing expedition.
• "Significant = practically important":
  • Statistical significance ≠ effect size. 0.001% improvement statistically significant হতে পারে কিন্তু useless.

(৫) Effect size — p-value-এর companion:

• Cohen's d: $(\mu_1 - \mu_2) / \sigma$।
• $d < 0.2$: small, $0.5$: medium, $> 0.8$: large.
• "Significant + meaningful effect size" — দু'টোই দরকার।

(৬) Multiple testing problem:

• ২০টি hyperparameter try করলে — $5\%$ false positive accidental.
• Solution: Bonferroni ($\alpha/k$), Benjamini-Hochberg (FDR)।
• Hyperparameter sweep-এ সাবধান।

(৭) Bayesian alternative:

• Bayes Factor: $P(D|H_1) / P(D|H_0)$।
• "How much more likely is $H_1$?"
• Direct probability, no $\alpha$।

(৮) Production-friendly checklist:

• Document baseline accurately.
• Multiple runs (≥10) — both models.
• Same test set, no peeking.
• Pre-register hypothesis (if research)।
• Report mean ± std + CI + p-value + effect size.
• Consider business impact, not just statistical.

(৯) Bangladesh tech context:

• "AI startup achieved 95% accuracy on Bangla NER" — কোন test set?
• Local benchmarks limited — hard to compare.
• Sajek/IndicNLP datasets — standardize.

(১০) The "Statistical Crisis":

• Replication crisis: many published findings can't be reproduced.
• ML reproducibility crisis (NeurIPS, ICML)।
• Solution: open-source code, multiple seeds, pre-registered analysis.

মূল উপলব্ধি: "Model A is better than Model B" — easy to claim, hard to prove. Real ML rigor — multiple seeds, pre-registered analysis, effect size, business validation. Senior ML engineer এই rigor maintain করেন — junior দের accuracy point comparison দিয়ে impress.

Bootstrap (Bradley Efron, ১৯৭৯) — modern statistics-এর সবচেয়ে important computational invention. সব evaluation-এ আজ default.

(১) মৌলিক ধারণা:

• Population unknown — কিন্তু sample যথেষ্ট representative.
• Sample-কে population-এর "approximation" ধরো।
• Sample থেকে resample = "synthetic samples"।
• প্রতিটিতে statistic compute → distribution.

(২) কেন কাজ করে — Glivenko-Cantelli theorem:

• Empirical distribution function (ECDF) → true CDF, $n \to \infty$।
• Bootstrap = sampling from ECDF.
• $n$ যথেষ্ট হলে — bootstrap distribution ≈ sampling distribution.
• Mathematical guarantee under regularity conditions.

(৩) Bootstrap variants:

• Basic Bootstrap: Resample with replacement, compute statistic.
• Percentile Bootstrap: Bootstrap distribution-এর 2.5%, 97.5% percentile = 95% CI.
• BCa (Bias-Corrected and accelerated): Skewness adjustment.
• Studentized Bootstrap: Better for small samples.
• Block Bootstrap: Time series-এ — correlation preserve.

(৪) কখন fail করে:

• Heavy-tailed distributions: Extreme values rare → bootstrap underestimate variance.
• Small sample (n < 30): ECDF poor approximation.
• Dependent data: Time series, spatial — naive bootstrap breaks.
• Discrete extremes: Min/max-এর bootstrap distribution unstable.
• Moment non-existent: Cauchy distribution-এর mean undefined.

(৫) Real ML production:

• Model evaluation: Test set accuracy-এর CI.
• Feature importance: Bootstrap permutation importance — Random Forest-এ।
• Bagging (Bootstrap Aggregating): Random Forest-এর foundation. Breiman ১৯৯৬।
• Hyperparameter sensitivity: Different bootstrap samples-এ best hyperparameter ভিন্ন? — instability indicator.
• A/B testing: Conversion rate-এর CI.
• Recommender system: Item ranking-এর uncertainty.
• Calibration: Predicted probability-এর reliability.

(৬) Implementation tips:

def bootstrap_ci(data, statistic, n_boot=10000, alpha=0.05):
    boot_stats = []
    n = len(data)
    for _ in range(n_boot):
        sample = np.random.choice(data, size=n, replace=True)
        boot_stats.append(statistic(sample))
    boot_stats = np.array(boot_stats)
    lower = np.percentile(boot_stats, 100 * alpha / 2)
    upper = np.percentile(boot_stats, 100 * (1 - alpha / 2))
    return lower, upper

# Median CI
ci = bootstrap_ci(data, np.median)
print(f"Median 95% CI: {ci}")

(৭) Bootstrap-এর computational cost:

• $n_{boot} = 10,000$ — adequate for percentile CI.
• Parallel — embarrassingly parallel.
• Modern CPU-এ ১ সেকেন্ডে done.
• GPU-এ vectorize ১০x faster.

(৮) Bagging — bootstrap-এর জাদু:

• Random Forest: $B$ bootstrap samples, প্রতিটিতে decision tree, vote/average.
• Variance reduction. Bias same.
• $B = 100-1000$ — typical.
• Out-of-bag (OOB) error — built-in cross-validation.

(৯) Modern variants:

• Bayesian bootstrap: Continuous weights instead of integer counts.
• Wild bootstrap: Heteroskedastic regression.
• Bootstrap aggregating in deep learning: Limited — too expensive.

(১০) Bangladesh ML scenarios:

• Bangla sentiment dataset — small, noisy. Bootstrap CI critical.
• BB টেস্ট ক্রিকেট data — bootstrap for player performance.
• Pathao/Foodpanda A/B test — bootstrap for conversion lift.
• SME loan default prediction — bootstrap importance ranking.

মূল উপলব্ধি: Bootstrap — সরল কিন্তু revolutionary. "Sample is the population in microcosm" — এই philosophical insight দিয়ে যেকোনো statistic-এর uncertainty বের। ১৯৭৯-এর invention আজকের ML production-এ default tool. Beautiful science, beautiful application.