Vanishing/Exploding Gradient

Sequence modeling-এর ৩০ বছরের struggle। Bengio-র ১৯৯৪ paper "Learning long-term dependencies with gradient descent is difficult" — RNN-এর fundamental challenge identify করে। LSTM (Hochreiter-Schmidhuber ১৯৯৭) ছিল breakthrough।

RNN-এর problem:

• Same weight $W$ repeatedly applied across time steps।
• Backprop through time (BPTT): $\frac{\partial h_T}{\partial h_0} = \prod_{t=1}^{T} \frac{\partial h_t}{\partial h_{t-1}}$।
• Each derivative ≈ $W \cdot \text{diag}(\sigma'(\cdot))$।
• Spectral radius of $W$ < 1 → exponential decay (vanishing)।
• Spectral radius > 1 → exponential growth (exploding)।
• Long sequence (100+ steps) — practically impossible।

LSTM-এর fix — cell state highway:

• Cell state $c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t$।
• Gradient: $\frac{\partial c_t}{\partial c_{t-1}} = f_t$।
• If forget gate $f_t \approx 1$ → gradient flow undisturbed।
• Linear path through cell state — bypasses non-linear saturation।

Gating intuition:

• Forget gate: "remember" or "forget" past।
• Input gate: "accept" new info।
• Output gate: "expose" to next step।
• Each learned, controlled by data।

GRU simplification:

• Cho et al. (২০১৪) — fewer gates।
• Update gate + reset gate।
• Cell state ও hidden merged।
• Empirically — LSTM-এর সমান performance, ফাস্ট।

Theoretical analysis:

• "Constant error carousel" — Hochreiter-এর term।
• Cell state via Identity-like derivative।
• Saturation isolated to gates, not the long-term path।
• Effective gradient flow ~ $\prod_t f_t$ — controllable।

Limits of LSTM:

• Very long sequence (1000+) still difficult।
• Sequential nature — hard to parallelize।
• Cumulative numerical error।
• Transformer-এর attention সব pair-এ direct connection — superior।

Transformer-এর approach:

• Self-attention: each position other positions-এ direct attention।
• Path length O(1), not O(T)।
• Vanishing fundamentally avoided।
• Position encoding — sequential info preserve।

Modern hybrid:

• RWKV — RNN-Transformer hybrid।
• State Space Models (S4, Mamba) — linear RNN with proper init।
• Long sequence 100K+ tokens।

Bangladesh context:

• Bangla NLP — Transformer-based now standard।
• BanglaBERT, IndicBERT — used widely।
• Long document Bangla — challenge active।

মূল উপলব্ধি: LSTM/GRU vanishing-এর partial solution, Transformer fundamental rethink। Architecture innovation often math-driven — vanishing analysis ছাড়া এই progress-গুলো হতো না। DL-এর progress = compute + algorithm + theory triad।

AI history-র সবচেয়ে interesting question। অনেক factors converge করেছিল।

Early ingredients (১৯৮০-২০০০):

• Backpropagation — Rumelhart-Hinton-Williams ১৯৮৬।
• Convolutional net — LeCun ১৯৮৯।
• LSTM — Hochreiter-Schmidhuber ১৯৯৭।
• RBM, Deep Belief Net — Hinton ২০০৬।

What was missing — ৫টি critical factor:

(১) Compute:

• ১৯৯০-এ training a 5-layer net = days।
• ২০১২-এ GPU = 1000x speedup।
• NVIDIA CUDA (২০০৭) — researcher-friendly।
• AlexNet trained on 2 GTX 580 — যা impossible ছিল CPU-তে।

(২) Data:

• ImageNet (Fei-Fei Li ২০০৯) — 1.4M labeled images।
• আগে — MNIST 60K, CIFAR 50K — too small for deep models।
• Internet-এর rise — large-scale data accessible।

(৩) Vanishing gradient solutions:

• ReLU (Glorot, Bordes, Bengio ২০১১)।
• Xavier init (Glorot ২০১০)।
• Batch Norm (Ioffe-Szegedy ২০১৫)।
• Residual (He ২০১৫)।
• Adam (Kingma-Ba ২০১৪)।

(৪) Software:

• Theano, Caffe (২০০৮-২০১৩)।
• TensorFlow ২০১৫, PyTorch ২০১৭।
• Reusable, GPU-aware, automatic differentiation।

(৫) Cultural/community shifts:

• ২০০০-এ NIPS reject neural net papers (almost)।
• SVM, Random Forest dominant।
• "AI winter" — neural net pejorative।
• Hinton, LeCun, Bengio persistent।

The trigger — AlexNet (২০১২):

• Krizhevsky-Sutskever-Hinton।
• ImageNet top-5 error 26% → 15% — massive improvement।
• Deep CNN + ReLU + dropout + GPU।
• Computer vision community shocked।

The cascade:

• ২০১২-২০১৫: CV revolution (VGG, GoogLeNet, ResNet)।
• ২০১৪-২০১৭: NLP catch up (Word2Vec, seq2seq, attention)।
• ২০১৭-২০২০: Transformer everywhere।
• ২০২০-now: LLM explosion।

Counterfactual — could it have happened earlier?

• Without GPU — no। Compute-bound।
• Without ImageNet — no। Data-bound।
• Without ReLU — slowly possible, much harder।
• Without funding/persistence — Hinton group could have given up।

The "bitter lesson" (Sutton ২০১৯):

• "General methods that leverage compute" — winners।
• Hand-crafted feature, knowledge — losers।
• Scale > cleverness — controversial but largely true।

Bangladesh implications:

• Compute access (cloud GPU) — equalizing factor।
• Bangla data scarcity — bottleneck।
• BUET, IUT-এ research possible এখন।
• ৩০ বছর আগে impossible — আজ student-এর reach-এ।

মূল উপলব্ধি: Deep learning — ১৯৮০-এর idea + ২০১০-এর infrastructure। Algorithm alone insufficient — compute + data + community + persistence required। Field-এর progress non-linear, threshold-driven। আজকের bottleneck identify করতে — কাল-এর breakthrough understand essential।

ResNet (He ২০১৫) DL-এর সবচেয়ে important architectural innovation-গুলোর একটি। Simple idea, profound impact।

The problem ResNet solved:

• Pre-ResNet: 20+ layer training degradation observed।
• Counterintuitive — deeper should be at-least-as-good।
• Cause: optimization difficulty, not overfitting।

The simple idea:

def residual_block(x, F):
    return x + F(x)

• $F$ — usually 2-3 conv layers।
• If $F = 0$ → identity → no harm।
• Network can "skip" if needed।

Multiple interpretations:

(১) Gradient highway:

• Backward pass: $\frac{\partial y}{\partial x} = 1 + \frac{\partial F}{\partial x}$।
• "+1" — direct gradient path।
• Even if $F$ small — gradient flow।

(২) Residual function easier:

• If true mapping ≈ identity → learn $F = 0$ easier than learning $F = \text{identity}$।
• Init close to optimum.
• Inductive bias toward smooth mappings।

(৩) Implicit ensemble (Veit ২০১৬):

• $n$-residual blocks → $2^n$ paths through network।
• Each path different depth।
• ResNet ≈ ensemble of shallow networks।
• Deletion of single block — minor effect (vs catastrophic in plain net)।

(৪) ODE interpretation (NeuralODE):

• $x_{t+1} = x_t + F(x_t)$ — Euler step of $\dot{x} = F(x)$।
• ResNet = discrete ODE।
• Continuous limit — Neural ODE (Chen ২০১৮)।
• Time-as-depth interpretation।

(৫) Loss landscape smoothing (Li ২০১৮):

• Filter visualization — ResNet loss landscape much smoother।
• Plain deep net — chaotic, full of barriers।
• Residual makes optimization landscape friendly।

(৬) Identity preservation (signal propagation):

• Shao et al. — signal propagation theory।
• Signal-to-noise preserved across depth।
• Without skip — signal degrade exponentially।

Practical impact:

• ResNet-152 — 152 layers, ImageNet SOTA।
• EfficientNet, Inception-v4 — all residual।
• Transformer block — residual standard।
• U-Net, segmentation — residual extensively।

Variations:

• DenseNet: all-to-all connections।
• Highway networks: learned gating।
• Stochastic depth: random layer drop।
• Pre-activation: BN before conv (cleaner)।

Transformer residual:

• Each sublayer: $\text{LayerNorm}(x + F(x))$ (post-LN)।
• OR: $x + F(\text{LayerNorm}(x))$ (pre-LN)।
• Pre-LN better for very deep — modern default।

Why "1" works geometrically:

• Without "1" — deep network's effective Jacobian product matters।
• With "1" — input always has direct path to output।
• Like adding express lane — local roads still exist, but throughway available।

মূল উপলব্ধি: ResNet — DL revolution-এর architecture innovation। Multiple interpretations valid — gradient, ensemble, ODE, landscape। Simple change, profound impact। Modern deep architecture — almost universally residual। Bangladesh-এ — যেকোনো architecture build-এ residual standard practice।

Modern Transformer training-এর common observation। Layer-wise gradient pathology — debug করা productive।

Observed pattern:

• Top (output-close) layer: gradient norm 100।
• Bottom (input-close) layer: 0.001।
• 5 orders of magnitude difference — 100,000x।
• Middle layers — gradual decline।

Possible causes:

(১) Standard residual + LayerNorm-এ post-LN bug:

• $y = \text{LN}(x + F(x))$ — original Transformer।
• Bottom layers see normalized signal — gradient compressed।
• Top layers see raw — large gradient।
• Solution: switch to pre-LN: $y = x + F(\text{LN}(x))$।

(২) Initialization scale issue:

• Standard init not optimal for very deep Transformer।
• T-Fixup (Huang ২০২০) — depth-aware initialization।
• DeepNorm (Wang ২০২২) — for 1000-layer Transformer।

(৩) Weight tying (embedding/lm_head):

• Embedding gradient accumulates from output and input use।
• Sometimes large values।

(৪) Loss landscape mismatch:

• Top layer learns mapping-specific transform।
• Bottom — low-level features, less to learn after pre-training।

Diagnosis tools:

# Per-layer gradient histogram
for name, p in model.named_parameters():
    if p.grad is not None:
        print(f"{name:40s} norm={p.grad.norm():.4e} max={p.grad.abs().max():.4e}")

Treatment strategies:

• (১) Pre-LN architecture:

  # Old (post-LN, original Transformer)
  y = LayerNorm(x + Attn(x))
  
  # New (pre-LN, GPT-style, modern default)
  y = x + Attn(LayerNorm(x))

• (২) Layer-wise learning rate (LARS-style):

  # Per-layer adaptive lr
  for layer_idx, layer in enumerate(model.layers):
      layer.lr_scale = 1.0 / (1 + layer_idx * 0.1)

• (৩) Discriminative fine-tuning (Howard-Ruder):
  • Bottom layers: low lr (already learned features)।
  • Top layers: high lr (task-specific)।
• (৪) Gradient clipping per parameter:

  torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
  # Or per-group clip

• (৫) DeepNorm initialization (very deep):

  # Wang et al. 2022
  alpha = (2N) ** 0.25  # N = layers
  beta = (8N) ** -0.25
  # Scale residual: x * alpha + F(x)
  # Init weights with beta scale

• (৬) Warmup schedule mandatory:
  • Sudden large lr at start → gradient explosion at top।
  • Linear warmup 1000-10000 steps।

Architecture-specific:

• BERT: post-LN, but small depth (12-24)।
• GPT-2/3: pre-LN — better stability।
• LLaMA: RMSNorm + pre-LN।
• PaLM: parallel sublayer।

Empirical observations:

• Pre-LN: bottom layers slowly learn — ok।
• Post-LN: bottom layers might not learn at all — bad।
• Modern best — pre-LN + warmup + clip।

Monitoring in production:

• Wandb/TensorBoard layer-wise gradient histograms।
• Alert if any layer gradient norm > 100 or < 1e-6।
• Track during training — early diagnosis।

Bangladesh practical:

• Bangla LLM fine-tune-এ pre-LN base model use।
• Layer-wise lr decay (LLRD) standard practice।
• Bottom embedding layers — very low lr (1e-5)।

মূল উপলব্ধি: Layer-wise gradient pattern — Transformer training-এর diagnostic tool। Pre-LN architecture + careful init + warmup → 5 orders of magnitude → 1-2 orders। Modern Transformer training tightly engineered। Default trust — layer-wise debug — issue catch।