Dropout — Overfitting থামানো

Dropout-এর nature — intuitive idea, evolving understanding। Original (২০১৪) paper-এ ensemble interpretation, পরে multiple alternatives।

Original ensemble view:

• $n$ neurons + dropout → $2^n$ possible sub-networks।
• Each forward — one sub-network sample।
• Inference — geometric mean of all।
• Implicit Bayesian model averaging।

Critique:

• $2^n$ sub-networks — but only $T$ training steps explored।
• Most sub-networks never sampled।
• Geometric mean approximation — approximate।
• Real "ensemble" দ্বারা produce করলে different result।

Bayesian perspective (Gal-Ghahramani ২০১৬):

• Dropout = approximate variational inference।
• Bayesian neural network-এ Bernoulli posterior।
• "MC Dropout" — multiple inference passes → uncertainty estimate।
• Practical uncertainty quantification।

Regularization view:

• Wager-Wang-Liang (২০১৩) — dropout ≈ specific L2 regularization (linear case)।
• Equivalent to adaptive penalty।
• Closed-form analysis available।

Information bottleneck view:

• Dropout — input compression/noise injection।
• Achille-Soatto framework।
• Generalization via information loss।

Co-adaptation prevention:

• Original Hinton motivation।
• Empirically true — drop-trained network feature interpretable।
• Each neuron useful independently।

Modern empirical observations:

• BN often replaces dropout effectively (CNN)।
• Transformer — dropout still essential।
• Reason — different architectures different overfitting modes।

Recent advances:

• Stochastic Depth: entire layers drop — extreme dropout।
• DropPath: Vision Transformer variant।
• R-Drop (২০২১): consistency between two dropout passes।
• Dropout in attention: different role than feature dropout।

Theoretical limits:

• Generalization bound (PAC-Bayes) — dropout improves।
• But bounds usually loose — practice exceeds theory।
• Active research area।

When dropout helps:

• Small dataset, large model।
• Overfitting evident (train-val gap)।
• Regularization-এর demand।
• Uncertainty estimation desired (MC Dropout)।

When dropout hurts:

• Already heavy regularization (BN + augmentation + weight decay)।
• Convergence speed critical (slows training)।
• Very deep CNN — Stochastic Depth better।

Bangladesh practical:

• Bangla NLP fine-tune — dropout 0.1 standard।
• Small medical dataset — dropout 0.3-0.5।
• Image classification — modern CNN (ResNet) usually no dropout।

মূল উপলব্ধি: Dropout — multiple valid interpretations। Ensemble/Bayesian/regularization — each captures aspect। None complete। Empirically robust, theoretically rich। Modern usage — domain-specific (NLP heavy, vision light)। Bangladesh-এ — overfitting symptom থেকে dropout decide।

DL-এর সবচেয়ে famous origin story-গুলোর একটি। Hinton multiple talks-এ retell করেছেন।

The bank visit:

• Hinton bank-এ গিয়েছিলেন।
• Tellers periodically rotate position।
• প্রশ্ন করলেন কেন এই rotation?
• Bank manager: "যাতে কেউ একটানা একই জায়গায় থেকে fraud-এর সুযোগ না পায়।"

Hinton-এর insight:

• Conspiracy needs collusion।
• Random rotation breaks conspiracy formation।
• Each individual must be honestly competent।
• Analog — neurons co-adapt for fragile features।
• Random drop forces each neuron individually useful।

The translation to neural nets:

• Neuron = bank teller।
• Co-adaptation = collusion।
• Random drop = position rotation।
• Robust feature = honest competence।

Story timeline:

• ২০১২ NIPS workshop — first dropout mention।
• ২০১২ ImageNet AlexNet — used dropout।
• ২০১৪ JMLR full paper — formal description।

The broader Hinton story-style approach:

• Hinton ML idea-গুলো analogy থেকে আনেন।
• RBM ↔ statistical mechanics।
• Capsule network ↔ visual perception theory।
• Forward-forward ↔ sleep cycles।

Why this style works:

• Real-world phenomena — robust, well-studied।
• Cross-domain inspiration — fresh perspective।
• Pedagogical advantage — explainable।
• Simple ideas often best।

Other DL ideas-এর origins:

• CNN: Visual cortex neuroscience (Hubel-Wiesel)।
• Attention: Human selective attention।
• Transformer: "Attention is all you need" — pure novelty (no analogy)।
• Reinforcement Learning: Behavioral psychology (Thorndike)।

Was Hinton's story exact?

• Probably embellished।
• Real research process iterative।
• Story popularity — pedagogical use।
• Inspirational regardless of literalness।

Lesson for researchers:

• Cross-domain reading — math, biology, economics, art।
• Analogies seed ideas।
• Try simple things first।
• Story-tell your work — adoption helped।

Bangladesh context:

• Local domain — Bangla, agriculture, traffic — unique inspirations।
• Dhaka traffic ↔ network routing problem।
• Rice cultivation ↔ optimization scheduling।
• Bengali poetry meter ↔ sequence pattern।

Hinton-এর bank teller analogy-র power:

• Memorable — once heard, never forgotten।
• Concrete — abstract idea grounded।
• Convincing — implementation follows naturally।
• "Aha moment" — pedagogy gold।

মূল উপলব্ধি: Best ideas often analogy-driven। Hinton's bank teller — DL pedagogy-র classic। Real research iterative + serendipitous + inspired। Bangladesh-এ — local culture-domain থেকে fresh analog খুঁজা — research opportunity। Story-telling research communicate-এ critical।

Gal-Ghahramani (২০১৬) "Dropout as a Bayesian Approximation" — DL uncertainty quantification-এর landmark।

Standard inference:

• Train with dropout।
• Eval-এ dropout off।
• Single deterministic prediction।
• Confidence — soft-max → may be wrong-confident।

MC Dropout idea:

• Train with dropout (normal)।
• Inference-এ dropout রাখো।
• Same input — multiple forward passes।
• Different sub-network → different prediction।
• Mean = prediction, variance = uncertainty।

Code:

def predict_with_uncertainty(model, x, n_samples=20):
    model.train()  # dropout on (yes, in inference!)
    preds = torch.stack([model(x) for _ in range(n_samples)])
    mean = preds.mean(0)
    var  = preds.var(0)
    return mean, var

# Usage
prediction, uncertainty = predict_with_uncertainty(model, x)
# High uncertainty → low confidence → human review

Theoretical foundation:

• Gal-Ghahramani: dropout NN ≈ Gaussian Process posterior।
• Bernoulli variational distribution over weights।
• MC samples ≈ Bayesian posterior samples।
• Variance ≈ epistemic uncertainty।

Two types of uncertainty:

• Epistemic (model uncertainty):
  • "What we don't know"।
  • Reducible with more data।
  • MC Dropout captures this।
• Aleatoric (data uncertainty):
  • Inherent noise in data।
  • Irreducible।
  • Need different modeling।

Production benefits:

• Active learning: uncertain samples — human label।
• Out-of-distribution detection: high uncertainty → flag।
• Safety-critical: medical, autonomous → reject if uncertain।
• Confidence calibration: better risk assessment।

Limitations:

• Inference cost — 20-100x slower।
• Quality — approximate Bayesian (not perfect)।
• Calibration — depends on dropout rate।
• Doesn't capture aleatoric well।

Alternatives for uncertainty:

• Deep Ensembles: multiple full models — best quality but expensive।
• SWAG: Stochastic Weight Averaging Gaussian।
• BNN proper: variational inference — complex।
• Conformal prediction: coverage guarantee।
• Temperature scaling: post-hoc calibration।

Empirical comparison:

• Deep Ensembles often best calibrated।
• MC Dropout — fastest principled approach।
• Temperature scaling — minimal cost, decent।

Bangladesh use cases:

• Medical AI: X-ray disease classification — uncertain → radiologist review।
• Agriculture: crop disease detection — uncertainty trigger expert visit।
• Bangla speech recognition: low-confidence — request human transcription।
• Fraud detection: uncertain transaction → manual review।

Production considerations:

• Latency budget — 20x inference acceptable?
• Better — train calibrated model + ensemble during deployment for critical cases।
• Hybrid — fast model + slow ensemble for low-confidence।

Modern trend — model trustworthiness:

• Anthropic, OpenAI — uncertainty in LLM responses।
• "Tell me what you don't know"।
• Calibration research increasing।

মূল উপলব্ধি: Dropout-এর hidden gem — uncertainty quantification। Single line code change — Bayesian-ish। Production-এ trust + safety। Bangladesh-এ medical/agriculture AI deploy-এ এই tool game-changer। Plain accuracy beyond — calibrated confidence-এর era।

Practical fine-tuning scenario। Decision factors many।

Problem assessment:

• 5000 samples — small for BERT (110M params)।
• 10% gap — moderate overfitting।
• Bangla — specific morphological richness।
• Need careful regularization।

BERT default dropout:

• Hidden dropout: 0.1।
• Attention dropout: 0.1।
• Both already present।

Strategy options:

(১) Increase dropout (light):

• Hidden 0.2, attention 0.1।
• Slight overfitting reduce।
• Don't break pre-training equilibrium।

(২) Increase dropout (aggressive):

• Hidden 0.3-0.5।
• Strong regularization।
• Risk — under-fit, slow convergence।

(৩) Combine techniques:

• Mild dropout increase: 0.2।
• Weight decay: 0.01।
• Early stopping: patience 3।
• Data augmentation: synonym, back-translation।

Recommended approach:

from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained(
    "csebuetnlp/banglabert",
    num_labels=3,
    hidden_dropout_prob=0.2,        # default 0.1
    attention_probs_dropout_prob=0.1,  # keep default
)

optim = AdamW(
    model.parameters(),
    lr=2e-5,
    weight_decay=0.01,
)

# Early stopping
from transformers import TrainingArguments
args = TrainingArguments(
    num_train_epochs=10,
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    greater_is_better=False,
    save_strategy="epoch",
    eval_strategy="epoch",
    early_stopping_patience=3,
)

Bangla-specific considerations:

• BanglaBERT, IndicBERT base preferred।
• Mixed Bangla-English — multilingual BERT।
• Tokenizer matters — sub-word for compound words।

Data augmentation for Bangla:

• Synonym replacement: Bangla wordnet (limited)।
• Back-translation: Bangla → English → Bangla।
• EDA (Easy Data Augmentation): random swap, delete।
• Mixup: embedding space mix।

Other regularizations:

• Label smoothing: 0.1 — confidence reduce।
• R-Drop: two dropout passes — KL divergence।
• Mixout: mix between fine-tuned and pre-trained weights।
• FreezeOut: early layers freeze gradually।

Validation strategy:

• 5-fold cross-validation — small data।
• Stratified sampling — class balance।
• Hold-out test — final evaluation।

Hyperparameter search:

• Sweep: dropout {0.1, 0.2, 0.3}।
• Sweep: lr {1e-5, 2e-5, 5e-5}।
• Sweep: weight_decay {0, 0.01, 0.1}।
• Optuna/Wandb sweep — automated।

If gap remains:

• More augmentation।
• Smaller model — DistilBERT (66M)।
• Multi-task learning — auxiliary objective।
• Domain pre-training — Bangla corpus।

If under-fits with high dropout:

• Reduce dropout 0.3 → 0.2।
• Increase capacity — bigger model।
• Longer training।
• Lower lr but more epochs।

Production considerations:

• Inference latency — dropout off (default eval)।
• Model size — DistilBERT for mobile।
• Serving — ONNX export।
• Monitoring — production drift detect।

Realistic expectations:

• 5000 samples — 75-85% accuracy reasonable।
• 10% gap → maybe 5% with regularization।
• Major improvement — more data।
• Active learning — efficient labeling।

Bangladesh data scarcity recipe:

• Pre-trained Bangla model।
• Light dropout increase (0.1 → 0.2)।
• Weight decay 0.01।
• Data augmentation aggressive।
• Cross-validation evaluate।
• Active learning iterate।

মূল উপলব্ধি: Fine-tuning small data + large pre-trained model = regularization optimization। Dropout single dimension — combination key। Bangla specifically — language-specific pre-training + augmentation valuable। 5000 sample-এ — engineering excellence + domain knowledge essential। 10% gap → 3-5% achievable but data scaling more impactful long-term।