Computational graph

AI framework history-র সবচেয়ে fascinating arc। ২০১৫-২০২০ TF dominant ছিল production-এ, PyTorch research-এ। এখন PyTorch দু'টোতেই lead। কারণ design philosophy-র বিবর্তন।

Static graph (TF 1.x)-এর rationale:

• Compiler-style optimization — kernel fusion, constant folding।
• Distributed training — partition automatic।
• Cross-platform deployment — mobile, web।
• Production-grade serving (TF Serving)।

Static graph-এর pain:

• Define-then-run — debug nightmare।
• Control flow বিকৃত — tf.cond, tf.while।
• Variable scoping এক চ্যালেঞ্জ।
• Researcher-দের জন্য productivity killer।

Dynamic graph (PyTorch)-এর win:

• Pythonic — control flow স্বাভাবিক।
• Print, breakpoint কাজ করে।
• Quick prototyping।
• Research community দ্রুত adoption।

Dynamic-এর cost:

• Per-iteration build overhead।
• Optimization opportunity miss।
• Hardware-specific code generation কঠিন।

Convergence: best of both:

• TF 2.x eager mode default — PyTorch-এর experience।
• tf.function decorator — selective static compilation।
• PyTorch torch.jit.script — early static option।
• torch.compile (২০২৩) — TorchInductor backend।
• JAX — pure functional, jit-compilable।

Lesson: trace-then-compile:

• Eager execution → trace operations → compile graph → run optimized।
• Best abstraction — user pythonic, system optimized।

Why PyTorch won research:

• Cleaner API — fewer abstractions।
• NumPy-like — familiar।
• Hugging Face, FastAI — ecosystem।
• Meta-র aggressive open source push।
• Academic citation pattern — researcher A use → researcher B follow।

Why production catching up:

• TorchServe, TorchScript — deployment।
• ONNX export — cross-framework।
• Mobile (PyTorch Mobile) — edge।
• torch.compile — production speed।

JAX-এর position:

• Functional pure — research-এ elegance।
• XLA compile — TPU-তে supreme।
• DeepMind, Google research preferred।
• Industry adoption সীমিত — learning curve।

Future trend:

• MLIR (Multi-Level IR) — universal compilation।
• Hardware-software co-design।
• OpenAI Triton — high-level kernel programming।
• Framework-agnostic models (HuggingFace adapters)।

মূল উপলব্ধি: "Static vs dynamic" false dichotomy। Modern framework — eager-by-default, compile-on-demand। Productivity ও performance দু'টোই pursue। Bangladesh-এ — PyTorch start, JAX explore, TF legacy deal — তিন framework ই জানা সমৃদ্ধ।

Computational graph শুধু DL-এর tool না — যেকোনো differentiable computation-এর foundation। "Differentiable programming" এই broad framework।

Differentiable programming concept:

• প্রোগ্রামকে এমনভাবে লেখা যাতে input-এর সাপেক্ষে output-এর gradient compute সম্ভব।
• "Software 2.0" — Karpathy-এর term।
• Backprop = differentiable programming-এর special case।

Application: Physics simulation:

• Differentiable physics simulator (DiffTaichi, Brax)।
• Soft body, fluid, rigid body — সবই differentiable।
• Robot policy directly through simulation gradient learn।
• Material property infer — observed deformation থেকে।

Application: Finance:

• Option pricing — Black-Scholes graph differentiable।
• Greeks (sensitivity) — auto-compute।
• Portfolio optimization — gradient-based।
• Risk model calibration — backprop through Monte Carlo।

Application: Robotics:

• Differentiable kinematics — joint angle থেকে end-effector।
• Inverse kinematics — gradient descent।
• Policy learning — sim-to-real।
• Motion planning gradient-based।

Application: Computer graphics:

• Differentiable rendering — image থেকে 3D recover।
• NeRF (Neural Radiance Fields) — volumetric rendering।
• Material/lighting estimation।
• Inverse graphics — vision-এর dual।

Application: Probabilistic programming:

• Pyro, NumPyro — Bayesian model।
• Variational inference — gradient-based।
• Stan-এর modern alternative।

Application: Scientific computing:

• Climate model gradient-aware।
• Differential equation solver — neural ODE।
• Quantum circuit optimization।
• Drug discovery — molecular property gradient।

JAX-এর role:

• Pure functional — composable transformations।
• jit, vmap, pmap, grad — primitive composability।
• NumPy API — scientific community familiar।
• DeepMind AlphaFold internal — JAX।

Bangladesh research opportunity:

• Cyclone simulation differentiable — Bangladesh climate research।
• Agriculture yield prediction — physics + ML hybrid।
• River dynamics modeling।
• Garment supply chain optimization।

Limitations:

• সব function differentiable না (discrete, conditional)।
• Soft relaxation often required।
• Gumbel-softmax, straight-through estimator — tricks।

মূল উপলব্ধি: Computational graph just DL tool নয় — universal computational paradigm। Differentiable programming আগামী দশকে scientific computing transform করবে। Just learning ML থেকে এটি বহু broader।

PyTorch-এর সবচেয়ে confusing error message-গুলোর একটি। বুঝতে — memory model-এ যেতে হবে।

Default behavior:

• backward()-এর পর graph-এর intermediate buffer free হয়।
• Memory save — common case (single backward per forward)।
• আবার backward() call → buffer নেই → error।

The classic error:

x = torch.tensor(2.0, requires_grad=True)
y = x ** 2

y.backward()      # OK — x.grad = 4
y.backward()      # RuntimeError: backward through graph second time

Solutions:

• Option 1: retain_graph=True

  y.backward(retain_graph=True)
  y.backward()      # works, x.grad = 8 (accumulated)

• Option 2: Recompute

  y = x ** 2
  y.backward()
  y = x ** 2        # rebuild
  y.backward()

কখন দরকার:

• Multiple losses, shared graph:

  shared = encoder(x)
  loss1 = head1(shared).mean()
  loss2 = head2(shared).mean()
  loss1.backward(retain_graph=True)
  loss2.backward()

  অথবা better: (loss1 + loss2).backward()।
• GAN training:

  # Discriminator update
  fake = G(z)
  d_loss = ...
  d_loss.backward()
  
  # Generator update — same fake!
  g_loss = ...
  g_loss.backward()    # error if same graph

  Solution: fake.detach() in D-loss।
• Higher-order gradients:

  y.backward(create_graph=True)  # auto-retain
  grad_x = x.grad
  grad2 = torch.autograd.grad(grad_x, x)[0]

• Custom training loops: meta-learning, MAML।

Pitfall — memory leak:

• retain_graph=True overuse → graph accumulate, OOM।
• Solution: explicit del graph_tensor বা scope-এ end।

Better patterns:

• Combine losses: (l1 + l2 + l3).backward() — single backward।
• Use autograd.grad instead of backward when partial gradient needed।
• detach aggressively — clear non-needed graph parts।

Debug strategy:

• Error হলে — কোন tensor বা loss double-backward হচ্ছে identify।
• Computation graph print: print(loss.grad_fn)।
• Hook দিয়ে gradient flow trace।

Performance impact:

• retain_graph — buffer keep, memory cost।
• create_graph — backward graph build, even more memory।
• Inference-এ no_grad mandatory — no graph at all।

মূল উপলব্ধি: Default behavior memory-efficient। retain_graph escape hatch — sparingly use। Better — graph design rethink (single combined loss, detach proper)। GAN, meta-learning-এ unavoidable। Beginner-দের জন্য — error দেখলে graph restructure আগে চেষ্টা করুন।

Modern LLM training-এর backbone technique। GPT-3, LLaMA — সবই checkpoint-এ নির্ভরশীল।

Memory problem:

• Backprop-এর জন্য সব activation save।
• 50-layer Transformer, 2048 seq, 4096 hidden — ভয়ংকর memory।
• Activation memory প্রায় $O(\text{layers} \times \text{batch} \times \text{seq} \times \text{hidden})$।

Standard approach without checkpoint:

• Forward — সব layer activation save।
• Backward — saved activation read, gradient compute।
• Memory: O(L) activations।

Gradient checkpointing core idea:

• Forward — কিছু "checkpoint" save, বাকি discard।
• Backward — discarded part recompute (forward again from checkpoint)।
• Memory ↓, compute ↑।

Math: optimal checkpointing:

• Naive: every layer checkpoint → memory O(L), no save।
• Sqrt strategy: $\sqrt{L}$ checkpoints → memory $O(\sqrt{L})$, compute $1.5\times$।
• Chen et al. (২০১৬) — recursive scheme, memory $O(\log L)$।

PyTorch implementation:

from torch.utils.checkpoint import checkpoint

class Block(nn.Module):
    def forward(self, x):
        # heavy computation
        ...

# Without checkpoint
y = block(x)

# With checkpoint
y = checkpoint(block, x, use_reentrant=False)
# Forward — block-এর activation drop
# Backward — block recompute, then gradient

Sequential checkpointing:

from torch.utils.checkpoint import checkpoint_sequential

# Auto-divide layers into segments
y = checkpoint_sequential(model.layers, segments=4, input=x)
# 4 checkpoints — memory O(L/4), compute 1.25x

Trade-off quantification:

• 50-layer, sqrt strategy → memory ÷ 7, compute × 1.5।
• Practically — 30-50% memory save, 25-35% compute overhead।
• Larger batch possible → better GPU utilization compensates।

When to use:

• OOM error during training।
• Want larger batch।
• Want longer sequence।
• Want bigger model।
• Compute budget allows।

When NOT to use:

• Memory enough — pure waste।
• Compute-bound, latency-sensitive।
• Inference (no backward — no need)।

Combination with other techniques:

• Mixed precision (FP16/BF16): activation memory ÷ 2।
• ZeRO (DeepSpeed): optimizer state shard।
• Tensor parallelism: activation across GPUs।
• FlashAttention: attention memory linear in seq।

Modern LLM training stack:

• Activation checkpointing + ZeRO + FP16 + Tensor parallel।
• 13B model train possible on consumer multi-GPU।
• 70B+ → cluster needed।

Bangladesh perspective:

• Limited GPU (Colab, RTX 3090) → checkpointing critical।
• Bangla LLM fine-tune (LLaMA-2 7B) → ১২GB GPU + checkpoint workable।
• Research lab — multi-GPU + DeepSpeed।

Pitfalls:

• Random ops (dropout) — recompute different result! Use seeded RNG।
• use_reentrant=False recommended (২.০+) — avoid hidden bugs।
• Profile carefully — সব time-এ benefit guaranteed না।

মূল উপলব্ধি: Gradient checkpointing — memory ↔ compute trade। Modern LLM-এ "must"। Combination with FP16, distributed training বহু large model-কে accessible করেছে। Memory-constrained scenario-তে first reach for this tool।