প্রত্যাশা ও ভেরিয়েন্স

Classical statistics ও modern DL-এর philosophical conflict — এই প্রশ্নে।

Classical bias-variance:

• $\text{Error} = \text{Bias}^2 + \text{Variance} + \text{Noise}$।
• Simple model: high bias (assumption rigid), low variance.
• Complex model: low bias, high variance (overfit)।
• Sweet spot: medium complexity → minimum total error.
• "U-shaped curve" — test error model complexity-এর সাথে।

"Modern" practice — DL contradicts:

• GPT-4 = 1.7T parameters, training data ~10T tokens.
• Massively overparameterized — variance অসম্ভব বেশি হওয়া উচিত।
• তবু — generalize অসাধারণ।
• Classical theory predicts disaster; reality smooth performance.

Double descent (২০১৯, Belkin et al.):

• Complexity বাড়ানোর সাথে test error ফিরে নামে — দ্বিতীয়বার।
• Classical regime: simple → moderate → over-parameterized (peak error)।
• Modern regime: over-parameterized → super-over-parameterized (error আবার কমে)।
• Double-descent curve — empirically প্রমাণিত।

কেন DL escape করে — possible explanations:

• Implicit regularization of SGD: SGD flat minima বাছে, সেগুলো generalize ভাল।
• Feature learning hypothesis: Big network = adaptive features. relevant features শিখে, irrelevant ignore.
• Lottery ticket hypothesis: Random init-এ বহু "winning subnetworks" — training সঠিকটি বাছে।
• Neural Tangent Kernel (NTK): অসীম-wide network — kernel regression-এর মতো behave. Tractable theory.
• Benign overfitting: Memorization without overfitting — possible high-dim-এ।

Practical implications:

• "Use fewer parameters than data" — old rule, now wrong.
• Modern: bigger almost always better, given data + compute.
• Scaling laws (Chinchilla) compute-optimal model size dictate.
• Fine-tuning small model → বড় model থেকে worse.

Caveats:

• Small data (e.g., medical) — classical regime এখনো applicable.
• Distributional shift — large models also fail.
• Theory-practice gap — open research.

মূল উপলব্ধি: "More parameters always overfits" — DL-এ ভুল। Bias-variance tradeoff মৃত নয়, কিন্তু DL-এ refined. আজও active research area.

BN — DL-এর একটি simple-অথচ-deep technique. ১০ বছর পেরিয়েও আজ মেকানিজম নিয়ে বিতর্ক।

মূল algorithm:

• Mini-batch-এ প্রতিটি activation channel-এ — mean ($\mu_B$) ও variance ($\sigma_B^2$) compute.
• $\hat{x} = (x - \mu_B) / \sqrt{\sigma_B^2 + \epsilon}$ — normalize.
• $y = \gamma \hat{x} + \beta$ — learnable scale + shift.
• Inference-এ — running mean/var ব্যবহার (training-এ updated)।

Original explanation (Ioffe & Szegedy, ২০১৫):

• "Internal Covariate Shift" — layer activations-এর distribution shift হয় training-এ।
• BN distribution stabilize করে।
• Faster training, higher learning rate possible.
• Implicit regularization (mini-batch noise)।

Modern challenges to original explanation:

• Santurkar et al. (২০১৮) — "How Does BN Help Optimization?": দেখায় internal covariate shift বাস্তবে BN-এর সাথে বাড়তে পারে। অথচ training improves.
• Real reason — loss landscape smoothing: BN gradient-এর Lipschitz constant কমায়, loss surface smoother হয়।
• Better gradient flow: Vanishing/exploding gradient এর চেয়ে কম sensitivity.
• Implicit regularizer: Mini-batch statistics noise → SGD-এর মতো effect.

Practical observations:

• Training 5-10x faster এ।
• Higher learning rates stable.
• Less sensitive to initialization.
• Mild regularization effect — dropout-এর সাথে redundant হতে পারে।
• Test-train mismatch (running stats use)।

Limitations:

• Small batch size (e.g., 1-4) → unreliable statistics.
• Sequence models (RNN, Transformer) — sequence length-এ batch dimension mismatch.
• Distributed training — batch synchronization overhead.

Alternatives evolved:

• Layer Normalization: per-sample, per-layer. Transformer-এ standard.
• Group Normalization: channel groups. small-batch-এ ভাল।
• Instance Normalization: per-sample, per-channel. Style transfer.
• Weight Normalization: weights-এর norm separate.
• RMSNorm: mean center skip. Llama, modern LLMs.

মূল উপলব্ধি: BN — "engineering hack that works"। Theory চলছে; practice mature. DL-এ এমন একটি common phenomenon — empirical সফলতা theoretical understanding-এর আগে।

এই প্রশ্ন AI-র সবচেয়ে গভীর research direction-গুলোর একটি ছোঁয়। Judea Pearl-এর "Causal Revolution" এর হৃদয়।

Standard ML — যা পারে:

• Pattern recognition: "যখন X ঘটে, Y-ও সাধারণত ঘটে।"
• Prediction: "X দেখে — Y predict."
• Anomaly detection.
• সবই correlation-based.

Causal reasoning — যা দরকার কিন্তু কঠিন:

• "যদি আমি X করি — Y কী হবে?" (intervention)।
• "X যদি না ঘটত — Y কী হতো?" (counterfactual)।
• Standard ML এই প্রশ্ন উত্তর দিতে পারে না।

Pearl-এর Ladder of Causation:

• Rung 1 — Association: "Roosters crow ⟹ sun rises"। Pure correlation. Standard ML/DL এই স্তরে।
• Rung 2 — Intervention: "Rooster silence করলে sun কি rise করবে?"। Cause-effect পরীক্ষা।
• Rung 3 — Counterfactual: "যদি আমি rooster না কিনতাম, my farm-এ আজ কী হতো?"। Most sophisticated.

Confounding variable example:

• Ice cream sales ↔ drowning. Both depend on temperature.
• Standard ML — strong correlation শেখে।
• Causal: ice cream ban করলে drowning কমবে না।

Causal AI methods:

• Causal graphs (DAGs): Nodes + arrows. Domain expert encode knowledge.
• Do-calculus (Pearl): "$P(Y | do(X))$" — intervention-এর math.
• Randomized control trials (RCT): Gold standard. Confounder eliminate.
• Instrumental variables: Confounder-এর effect isolate.
• Causal discovery algorithms: PC, FCI — data থেকে DAG infer.

Modern AI causal efforts:

• Microsoft DoWhy: Python causal inference framework.
• CausalML (Uber): uplift modeling.
• RL agents: Naturally interventional — actions take.
• LLM + reasoning: Chain-of-thought causal-like কিন্তু provably না।

Future directions:

• Causal representation learning — features-এ causal structure encode.
• Foundation models + causal — Yann LeCun, Bengio's vision.
• World models — environment-এর causal mental model.

মূল উপলব্ধি: Pure DL — Rung 1-এ আটকা। AGI-র জন্য Rung 2-3 দরকার। Causal AI = একটি active frontier; সম্ভব, কিন্তু challenging. "AI-এর হৃদয়ে statistics, কিন্তু true intelligence-এ causality."

এই প্রশ্ন — point estimate ও uncertainty quantification-এর সবচেয়ে practical clash.

"Expected 90%" — যা বলে না:

• Variance: Test runs-এ কত fluctuation? σ = ১% কি ১০%?
• Subgroup performance: Some demographic-এ ৭০%, অন্যদের ৯৭% — গড় ৯০%।
• Distribution shift: Training data-র মতো না হলে — ৯০% guarantee নেই।
• Confidence calibration: মডেল কি জানে যে কখন uncertain?

Variance-এর importance:

• Model A: 90% accuracy, σ = 0.5% — predictable, deployable.
• Model B: 90% accuracy, σ = 8% — sometimes 82%, sometimes 98%। Risky.
• Same expected value — completely different deployment risk.

Real-world disasters:

• Knight Capital (২০১২): Trading algorithm ৯৯.৯% time fine, এক bad scenario-এ $440M loss. Variance ignore.
• IBM Watson Health: "Cancer diagnosis accurate" — কিন্তু certain hospitals-এ poorly perform. Distribution shift.
• Tesla Autopilot edge cases: 99% safe, 1% catastrophic failure mode.
• COMPAS (criminal justice AI): Average accuracy fine, demographic disparities huge.

Confidence interval — কী দেয়:

• "95% CI: [88.5%, 91.5%]" — ভাল। Deployment-এ confident.
• "95% CI: [78%, 98%]" — wide. Production-এ unpredictable.
• Bootstrapping দিয়ে compute করা যায় — multiple test resamples.

Modern uncertainty quantification:

• Predictive variance: Bayesian DL — multiple weight samples → output distribution.
• MC Dropout (Gal, ২০১৬): Test time-ও dropout enabled — variance estimate.
• Deep Ensembles: Multiple models train, predictions average + variance.
• Conformal prediction: Distribution-free uncertainty bounds.
• Calibration techniques: Platt scaling, temperature scaling.

Deployment best practices:

• Multiple test sets: Variance বুঝতে — different distributions.
• Subgroup analysis: Demographic, geographic, temporal — gaps খুঁজুন।
• Worst-case bounds: 95% confidence-এর "minimum" performance.
• Monitoring in production: drift detection, performance alerts.
• Reject options: low-confidence prediction-এ "I don't know" বলা।

মূল উপলব্ধি: "Average accuracy ৯০%" — necessary but not sufficient. Production AI-তে variance + worst-case + calibration সব দরকার। Engineering wisdom: "trust the variance, not the mean."