Batch Normalization

DL-এর philosophical core প্রশ্ন। "Black box" সমালোচনা এর সাথে যুক্ত।

BN case:

• Original (২০১৫): "covariate shift reduce" — convincing narrative।
• ২০১৮: Santurkar et al. — actually loss landscape smoothes।
• ২০২০+: still debate — multiple effects।
• Method works regardless of explanation।

Other examples in DL:

• Dropout: "ensemble" interpretation incomplete। Real benefit perhaps regularization via noise।
• Adam: empirically supreme, theoretically flawed (AMSGrad fix barely matters)।
• Skip connections: multiple competing explanations।
• Attention: "soft alignment" — but more than that empirically।
• LayerNorm placement: pre-LN vs post-LN — empirical without clear theory।

The "alchemy" critique:

• Rahimi-Recht (২০১৭) NeurIPS test-of-time talk।
• "Machine learning has become alchemy"।
• Tricks accumulate without principled understanding।

Counter-argument — engineering vs science:

• Bridge engineering preceded structural mechanics theory।
• Steam engine before thermodynamics।
• Empirical success → theoretical understanding follow।
• Both valid endeavors।

Implications for practitioners:

• Trust empirical evidence: proven techniques use।
• Skeptical of explanations: "why" might be wrong even if "what" right।
• Ablate carefully: remove one variable, observe effect।
• Replicate across tasks: robust trick > one-paper wonder।

Implications for researchers:

• "Explanation" — hypothesis, not conclusion।
• Multiple explanations possible — Occam's razor + experiment।
• Theoretical analysis valuable even if method already works।
• "Understanding" deeper than performance।

Recent push toward "Mechanistic interpretability":

• Anthropic, DeepMind active research।
• What does each neuron compute?
• Circuits within Transformer।
• Gradient pathway visualization।

Modern field maturing:

• Loss landscape theory (Li et al.)।
• Neural Tangent Kernel।
• Information bottleneck (controversial)।
• Optimization theory (convergence rate analysis)।
• Gap between practice and theory narrowing।

Bangladesh perspective:

• Theory research-এ contribute possible।
• Empirical engineering-ও valid pursuit।
• Both scientific endeavors।
• Don't be ashamed of either।

Personal disposition:

• Engineer mindset: "does it work?"
• Scientist mindset: "why does it work?"
• Best researchers — both। Hopper, Feynman models।

মূল উপলব্ধি: DL-এ explanation lag practice-এর চেয়ে। "Why" matters but separate from "what works"। BN-এর mystery — DL-এর recurring theme। Maturity = practice trust + theory pursue + skeptical of both। Bangladesh-এ — both engineer ও scientist developmentally important।

Architecture ও normalization-এর pairing — DL design pattern।

BN in CV:

• Image batch — fixed-size, dense।
• Spatial dimensions repeat statistics meaningful।
• Channel-wise norm — feature map normalization।
• Batch size 32-256 typical — statistics reliable।

BN issues in NLP:

• Variable sequence length: padding affects statistics।
• Small batch: Transformer often batch 4-32, mini-batch noise high।
• Sequence position matters: averaging across positions destroys structure।
• Online inference: single sample — batch stats undefined।

LayerNorm advantages for NLP:

• Per-sample normalization — batch-independent।
• Sequence length variation — irrelevant।
• Single-sample inference — works trivially।
• Position information preserved।

RNN特殊 case:

• Time-step varying — BN need separate stats per step।
• Memory inefficient।
• LayerNorm — per-time-step, no issue।
• Cooijmans et al. (২০১৬) recurrent BN attempt — complex, rarely used।

Math difference:

• BN: normalize across batch & spatial, per channel।
• LN: normalize across all features, per sample।
• $\mu_{\text{BN}} = \mathbb{E}_{N,H,W}[x]$ — per channel।
• $\mu_{\text{LN}} = \mathbb{E}_{C,H,W}[x]$ — per sample।

Modern LLM uses RMSNorm:

• $y = \gamma \cdot x / \sqrt{\text{mean}(x^2) + \epsilon}$।
• Mean subtraction skip — faster।
• LLaMA, PaLM, Gemma — all RMSNorm।
• 5-15% inference speedup।

Vision Transformer crossover:

• ViT (Dosovitskiy ২০২০) — uses LayerNorm।
• Even in CV, Transformer uses LN।
• Not pure "domain" split — architecture-driven।

Hybrid approaches:

• ConvNeXt — modern CNN, uses LayerNorm।
• Swin Transformer — LayerNorm।
• Convergence happening।

GroupNorm — middle ground:

• BN doesn't work well, LN-ও suboptimal কখনো।
• GN — channels groups-এ split।
• Detection (Mask R-CNN), small-batch CNN — GN preferred।

SyncBN for distributed training:

• Per-GPU batch small (e.g., 2)।
• Sync stats across GPUs — full effective batch।
• Medical imaging — common pattern।

Practical recipe:

• CNN, batch ≥ 32: BN।
• CNN, batch < 16: GroupNorm।
• Transformer (any): LayerNorm/RMSNorm।
• RNN/LSTM: LayerNorm।
• GAN: InstanceNorm/LayerNorm।

Bangladesh practical:

• Bangla ASR/NMT — LayerNorm Transformer।
• X-ray classification — BN ResNet।
• Bangla TTS — InstanceNorm।

মূল উপলব্ধি: Normalization choice — architecture + batch size + task matter। "BN for CV, LN for NLP" — outdated rule। Architecture-driven (CNN → BN, Transformer → LN)। Modern best — task থেকে normalization derive। RMSNorm momentum building — efficiency focus।

GAN training notoriously unstable, BN একটি contributing factor।

GAN structure:

• Generator: latent z → image।
• Discriminator: image → real/fake।
• Adversarial training — equilibrium difficult।

BN-এর problems in GAN:

• (১) Sample correlation:
  • BN — same batch-এর samples-এর statistics-এ depend।
  • One sample-এর output other samples affect।
  • Independence assumption violated।
• (২) Train-eval mismatch:
  • Generation-এ usually batch 1।
  • Eval mode running stats — train mismatch।
  • Generated quality varies।
• (৩) Mode collapse contribution:
  • BN-এর batch dependency — generator sees similar samples in batch।
  • Batch all "memorize" same mode।
  • Diversity lost।
• (৪) Adversarial dynamics:
  • Discriminator stats shift quickly।
  • Running stats unreliable।

Alternatives in GAN:

• Instance Norm (Ulyanov ২০১৭):
  • Per-sample, per-channel.
  • Sample independence preserved।
  • Style transfer-এ pioneered।
• Spectral Norm (Miyato ২০১৮):
  • Discriminator-এর Lipschitz constraint।
  • Weight matrix-এর largest singular value normalize।
  • WGAN-GP-এর alternative।
  • SOTA GANs (BigGAN, StyleGAN) — used।
• Group Norm:
  • Batch-independent।
  • Mode collapse less severe।
• No normalization:
  • Some recent GAN — careful init + careful schedule।
  • Less common।

StyleGAN advancements:

• StyleGAN (Karras ২০১৯) — adaptive instance norm।
• $\text{AdaIN}(x, y) = \sigma(y) \cdot \frac{x - \mu(x)}{\sigma(x)} + \mu(y)$।
• Style feature $y$-এর scale ও shift apply।
• StyleGAN2 — modulate-demodulate alternative।

Diffusion model trend:

• Modern image generation — diffusion replaces GAN।
• U-Net + GroupNorm + attention।
• BN issues bypassed।
• Stable Diffusion, Imagen, DALL-E।

Mode collapse — broader issues:

• Generator mapping concentrate to few modes।
• Diversity lost।
• Multiple causes: architecture, loss, optimization, normalization।

Fixes (besides normalization):

• WGAN-GP: better loss।
• Mini-batch discrimination: diversity reward।
• Two-time-scale update rule (TTUR): different lr।
• Self-attention (SAGAN): long-range dependency।

Practical GAN normalization choice:

• DCGAN style: BN in generator, none in discriminator।
• BigGAN: SpectralNorm in discriminator, conditional BN in generator।
• StyleGAN: AdaIN/modulation।
• Conditional GAN: Conditional BN/IN।

Bangladesh GAN research:

• Bangla handwriting generation — DCGAN-style।
• Image super-resolution — perceptual + adversarial।
• Diffusion model preferred for new projects।

মূল উপলব্ধি: BN-এর batch dependency — GAN-এর adversarial dynamic-এ harmful। InstanceNorm/SpectralNorm/AdaIN — domain-specific solution। Modern image gen — diffusion shift, GAN issues sidestep। Architecture choice carefully consider — "always use BN" দশা wrong।

Common BN-related fine-tuning bug। Subtle but impactful।

Problem identification:

• Train ও val accuracy gap বিশাল (95% vs 60%)।
• Pre-trained model — usually generalize ভাল।
• Suspect overfitting + BN issue।

Possible BN-specific causes:

• (১) Running stats overwritten:
  • Pre-trained running_mean/var ImageNet-এ tuned।
  • Fine-tune-এ small dataset → running stats domain-specific override।
  • Train mode-এ stats update — original lose।
  • Eval-এ updated stats use — domain narrow → poor generalization।
• (২) Train-eval distribution shift:
  • Train batch — same domain।
  • Test sample — slightly different।
  • Running stats train-domain-specific।
  • Test prediction shift।
• (৩) Small batch fine-tuning:
  • Batch 8-16 — BN stats noisy।
  • Running average corrupt।

Fixes:

• (১) Freeze BN during fine-tune:

  def freeze_bn(model):
      for m in model.modules():
          if isinstance(m, (nn.BatchNorm1d, nn.BatchNorm2d, nn.BatchNorm3d)):
              m.eval()  # eval mode — running stats freeze
              for param in m.parameters():
                  param.requires_grad = False
  
  freeze_bn(model)

  ImageNet stats preserve, only conv weights tune।
• (২) Use GroupNorm fine-tuning:
  • BN-কে GN-এ replace (sometimes works)।
  • Distribution shift independent।
• (৩) Lower BN momentum:

  for m in model.modules():
      if isinstance(m, nn.BatchNorm2d):
          m.momentum = 0.01  # default 0.1, slower update

  Running stats slowly update — original mostly preserved।
• (৪) Regularization:
  • Dropout add।
  • Stronger augmentation।
  • Weight decay increase।
• (৫) Test-time adaptation:
  • Recent technique — test domain-এ BN stats adapt।
  • TTA, TENT etc।

Diagnosis steps:

• Compare train-mode val accuracy vs eval-mode val accuracy।
• Both — train-eval mismatch।
• Save running_mean/var before/after fine-tune compare।
• Per-layer activation distribution visualize।

Hugging Face Transformer comparison:

• Transformer LayerNorm — no running stats।
• Fine-tuning issue absent।
• One reason LN preferred — fine-tuning robust।

Best practices for transfer learning:

• (১) Linear probing first: all backbone freeze।
• (২) Gradual unfreezing: top layers first।
• (৩) BN freeze: almost always good idea।
• (৪) Discriminative lr: per-layer।
• (৫) Augmentation: domain-appropriate।
• (৬) Early stopping: overfitting catch।

Code recipe:

import torchvision.models as models

model = models.resnet50(pretrained=True)

# (1) Replace head
model.fc = nn.Linear(2048, num_classes)

# (2) Freeze BN
for m in model.modules():
    if isinstance(m, nn.BatchNorm2d):
        m.eval()
        m.weight.requires_grad = False
        m.bias.requires_grad = False

# (3) Lower lr for backbone
optim = AdamW([
    {'params': model.fc.parameters(), 'lr': 1e-3},
    {'params': [p for n, p in model.named_parameters() if 'fc' not in n], 'lr': 1e-4},
], weight_decay=1e-4)

# (4) Forward, careful train()/eval() switch

Bangladesh medical imaging context:

• Limited dataset (1000-10000 X-rays)।
• ImageNet-trained ResNet — strong starting point।
• BN freeze + low lr fine-tune — best practice।
• 5-10% accuracy improvement থেকে।

মূল উপলব্ধি: Fine-tuning-এ BN — silent issue। Default behavior — running stats override — small dataset-এ harmful। Freeze BN almost always helpful। Modern alternative — Transformer + LayerNorm — issue absent। Practitioner-এর mental checklist-এ "BN frozen?" essential।