Adam ও AdamW

ML history-র সবচেয়ে famous "embarrassing fix"। ICLR ২০১৮ best paper Reddi-Kale-Kumar এর "On the convergence of Adam and beyond"।

The flaw:

• Adam-এর convergence proof (২০১৪) assumed $\hat{v}_t$ monotone increasing।
• Counterexample: $\hat{v}_t$ can decrease since EMA of $g^2$।
• Result — Adam-এর effective lr non-monotone।
• Specific cases-এ converge না বা wrong direction।

Toy counterexample:

• Synthetic problem: $f_t(w) = C w$ যদি $t \mod 3 = 1$, else $-w$।
• Optimum: $w = -1$।
• Adam — $w = +1$-এ converge!
• Vanilla SGD ঠিকভাবে converge।

AMSGrad fix:

• $\hat{v}_t = \max(\hat{v}_{t-1}, v_t)$ — monotone enforce।
• Convergence proof now valid।
• Theoretical guarantee restored।

Why AMSGrad didn't take over:

• Empirical performance: sometimes worse than Adam।
• Practical issue rare: real DL training-এ counterexample-like dynamic uncommon।
• Memory overhead: additional $\hat{v}_{\max}$ store।
• Adam habit: ecosystem already established।
• Other fixes: RAdam, AdaBound — alternatives।

Recent perspective:

• Bock et al. (২০১৯) — flaw গুরুত্বপূর্ণ but practical impact small।
• Modern variants (AdamW, AMSGrad) acceptable।
• Theoretical hygiene matters for research, less for production।

Lessons:

• "Most cited paper" → most scrutinized → flaws found।
• Theoretical proof বিবরণ critical — assumption check।
• Empirical success ≠ theoretical correctness।
• Fix-করা paper নিজেই influential।

Other Adam variants:

• RAdam: rectified Adam — early-step variance issue address।
• AdaBelief: $v$ uses $(g - m)^2$ instead of $g^2$।
• NovoGrad: NVIDIA's normalize-first optimizer।
• Lion (২০২৩): sign of momentum — memory ÷2।
• Sophia (২০২৩): diagonal Hessian estimate।

মূল উপলব্ধি: Adam flawed-but-works। AMSGrad correct-but-rarely-better। Practical DL "good enough" theoretical engineering-এর চেয়ে important। Research-এ correct theory, production-এ tested practice — দু'টোই value।

সাত বছর ধরে DL community-তে hidden bug। Loshchilov-Hutter ২০১৭-এ explicitly identify।

SGD case — equivalent:

• L2 reg: loss = base + $\frac{\lambda}{2}\|w\|^2$।
• Gradient: $g + \lambda w$।
• SGD update: $w \leftarrow w - \eta(g + \lambda w) = (1 - \eta\lambda) w - \eta g$।
• Direct decay: $w \leftarrow w - \eta g - \eta \lambda w$ — exactly same।

Adam case — different:

• L2 reg: gradient $g + \lambda w$ → fed into $m, v$।
• $m = \beta_1 m + (1-\beta_1)(g + \lambda w)$।
• $v = \beta_2 v + (1-\beta_2)(g + \lambda w)^2$।
• Update: $\hat{m}/\sqrt{\hat{v}}$ — weight decay scaled by $1/\sqrt{v}$।
• Large gradient parameter — small effective decay।
• Small gradient parameter — large effective decay।

AdamW fix:

• Decay applied directly: $w \leftarrow w - \eta(\hat{m}/\sqrt{\hat{v}}) - \eta\lambda w$।
• Decay decoupled from adaptive scaling।
• Each parameter decays uniformly।

Why this matters:

• Adam + L2 — large weight grow disproportionately।
• AdamW — proper regularization।
• Generalization-এ measurable improvement।

Empirical evidence:

• Loshchilov-Hutter — CIFAR-100, ImageNet improvement।
• BERT (Devlin ২০১৮) — explicitly AdamW।
• Hugging Face default AdamW।
• Modern Transformer training — AdamW universal।

PyTorch implementation difference:

# Adam with weight_decay (BUGGY for adaptive)
optim.Adam(params, weight_decay=0.01)

# AdamW (correct)
optim.AdamW(params, weight_decay=0.01)

# Math difference:
# Adam:  g_eff = g + λw, then adaptive
# AdamW: w -= adaptive(g) + ηλw

Hyperparameter difference:

• Adam-এর effective weight decay-অনিয়মিত — λ harder to tune।
• AdamW-এর λ direct interpretation — easy to tune।
• AdamW-এ typical λ = 0.01-0.1, Adam-এ ভিন্ন scale।

Why hidden so long:

• Default weight_decay=0 Adam-এ — issue invisible।
• Vision-এ SGD dominant — Adam-এর regularization rarely tested।
• Empirical tuning hide subtle issues।

Ecosystem migration:

• PyTorch 1.0 — separate AdamW class।
• TF 2.x — AdamW in addons।
• JAX/Flax — Optax library।

Modern recommendation:

• Always use AdamW (not Adam) when weight_decay > 0।
• Vision: AdamW, weight_decay=1e-4।
• Transformer: AdamW, weight_decay=0.1।
• Fine-tune: AdamW, weight_decay=0.01।

মূল উপলব্ধি: Subtle math, big practical impact। DL field-এ এমন hidden bug research-এ দেখা যায়। Loshchilov-Hutter-এর insight — careful theoretical analysis production-এ pay off। আজকের সব major LLM (GPT, LLaMA, Claude) — AdamW-এ trained।

DL community-র সবচেয়ে famous one-liner। ২০১৬-এ Karpathy এই tweet করেছিলেন — joke ছিল কিছুটা, কিন্তু empirically ভাল default।

Why 3e-4 (≈ 0.0003):

• Empirical sweet spot: CV, NLP, RL — most cases-এ converge।
• Default-এর জন্য balance: too small slow, too large divergence।
• Adam-এর adaptive scaling: per-parameter normalization-এ scale-invariant।

Math behind 3e-4:

• Adam-এ effective per-step update: $\eta / \sqrt{\hat{v}}$।
• $\hat{v} \approx g^2$ — typical gradient ~0.1, $\sqrt{\hat{v}} \approx 0.1$।
• Effective: $0.0003 / 0.1 = 0.003$ — reasonable parameter step।
• Vanilla SGD-এ same effect পেতে $\eta \approx 0.01$।

When 3e-4 works:

• Standard architectures (CNN, RNN, Transformer)।
• Standard initialization।
• Mid-size models (10M-1B parameters)।
• Standard datasets (CIFAR, ImageNet, GLUE)।
• From-scratch training।

When to use lower lr:

• Fine-tuning pre-trained: 1e-5 to 5e-5।
  • BERT fine-tune: 2e-5।
  • GPT fine-tune: 1e-5।
  • Vision fine-tune: 1e-4।
• Very large model: 1e-4 (LLaMA 70B)।
• Sensitive layers: embeddings often need lower lr।
• Late-stage training: cosine decay → 0।

When to use higher lr:

• Small model + small data: 1e-3 to 3e-3।
• Linear probing: last layer only — 1e-2।
• RNN-এর recurrent weight: sometimes 1e-3।

Layer-wise lr (sometimes):

• Pre-trained backbone — low lr।
• New head — high lr।
• "discriminative fine-tuning" — Howard-Ruder (২০১৮)।

LR scheduling:

• Cosine decay: $\eta_t = \eta_0 (1 + \cos(t\pi/T))/2$ — Transformer-এ universal।
• Warmup: first 1-10% steps — linear ramp।
• Step decay: /10 at fixed epochs — old-school CV।
• OneCycle: Smith ২০১৭ — start small, peak, decay।

LR finder (Smith ২০১৭):

• 1e-7 থেকে 10 পর্যন্ত exponential sweep।
• Loss curve plot।
• Steepest descent-এর just before peak — optimal।

Modern LLM training:

• GPT-3 175B: 6e-5।
• LLaMA-2 70B: 1.5e-4।
• BERT base: 1e-4।
• Vision Transformer: 1e-3।

Practical wisdom:

• Start with 3e-4, scan 1e-4 to 1e-3 if time।
• Plot loss vs lr — pick steepest stable region।
• Always add warmup for stability।
• Always add decay for final convergence।

মূল উপলব্ধি: 3e-4 — empirical Schelling point, math-এ explainable। Standard scenario-এ default ভাল। Domain ও model size-এ adjust করতে হবে। Karpathy joke serious truth — most experimentation 3e-4 থেকে শুরু করা wise।

Modern LLM fine-tuning-এর কেন্দ্রীয় challenge। Multiple memory-efficient techniques।

The memory math:

• 7B parameters × FP32 (4 bytes) = 28 GB weights।
• Adam optimizer states (m, v): 2 × 28 = 56 GB।
• Gradients: 28 GB।
• Activations: variable (10-50 GB)।
• Total: 122-172 GB — A100 80GB এ অসম্ভব।

Solution category 1: Quantize optimizer states

• 8-bit Adam (Dettmers ২০২২): $m, v$ in INT8 → 4x save।
• bitsandbytes.optim.AdamW8bit।
• Quality near-identical to FP32 Adam।
• 32 → 8 bit accumulator → trick।

Solution category 2: Mixed precision

• Weight FP32 master copy, computation FP16/BF16।
• Memory ÷ 2 for activations।
• torch.cuda.amp automatic।
• BF16 over FP16 (better range)।

Solution category 3: Quantization

• QLoRA (Dettmers ২০২৩):
  • Base model 4-bit (NF4)।
  • LoRA adapter 16-bit।
  • 7B fits in 6 GB।

Solution category 4: Parameter-efficient fine-tuning

• LoRA (Hu ২০২১):
  • $W = W_0 + BA$ (B, A small rank)।
  • Only $B, A$ train — original frozen।
  • ~0.1% parameters trainable।
  • Adam state on these only।
• Prefix tuning, P-tuning: only soft prompts।
• Adapters: small inserted modules।

Solution category 5: ZeRO / FSDP

• Optimizer states sharded across GPUs।
• ZeRO-1: optimizer state shard।
• ZeRO-2: + gradient shard।
• ZeRO-3: + parameter shard।
• DeepSpeed, PyTorch FSDP।

Solution category 6: Gradient accumulation

• Small per-step batch।
• Multiple steps accumulate before update।
• Memory friendly, longer training।

Solution category 7: Activation checkpointing

• Discussed in L10।
• Activation memory ÷ 2-4।
• 30% extra compute।

Solution category 8: Lighter optimizer

• SGD + momentum: only 1x parameter overhead।
• Lion: only momentum, no second moment — half Adam memory।
• Sophia: diagonal Hessian — different trade-off।

Practical recipe for 7B on consumer GPU (24 GB):

• QLoRA + 4-bit base model।
• BF16 mixed precision।
• Gradient accumulation 8x।
• Activation checkpointing।
• LoRA rank 16-64।
• Hugging Face PEFT library।

Code example:

from peft import LoraConfig, get_peft_model
from transformers import BitsAndBytesConfig

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4")
model = AutoModelForCausalLM.from_pretrained("llama2-7b", quantization_config=bnb)

lora = LoraConfig(r=16, target_modules=["q_proj", "v_proj"])
model = get_peft_model(model, lora)

# 7B model — only ~80M params trainable
optim = bnb.optim.AdamW8bit(model.parameters(), lr=1e-4)

Bangladesh perspective:

• Consumer GPU (RTX 3090/4090) — QLoRA enables 13B fine-tune।
• Free Colab — LoRA on Llama 7B।
• Bangla LLM fine-tune very accessible now।

মূল উপলব্ধি: Memory hierarchy DL training-এ central। 7B model accessible — পাঁচ বছর আগে impossible। Quantization + parameter-efficient + sharding combination revolutionary। Bangladesh-এ Bangla LLM build করা — এই tools-এ democratized। যথাযথ প্রয়োগ-এ — modest hardware-এ frontier capability।