Forward pass — ছবি থেকে অনুমান

আজকের LLM economics-এর কেন্দ্রে এই প্রশ্ন।

GPT-4 size estimate:

• Parameter count: ~১.৭৬ trillion (rumored, MoE architecture)।
• Active parameters per token: ~২২০ billion।
• Per token forward pass: ~৪৪০ billion FLOPs (২x parameters)।

একটি token-এর journey:

• ১. Input embedding lookup।
• ২. Position embedding যোগ।
• ৩. ১২০+ Transformer blocks — প্রতিটিতে attention + MLP।
• ৪. Final linear layer — vocabulary projection (50K+ tokens)।
• ৫. Softmax + sample।

Sequential nature:

• Token $t$-এর জন্য — token $1, 2, \ldots, t-1$ লাগে।
• Parallel possible না autoregressive generation-এ।
• ১০০ token output = ১০০ sequential forward pass।

KV cache optimization:

• Past tokens-এর Key, Value state cache।
• প্রতি new token-এর জন্য — শুধু সেই token-এর Q compute।
• Massive speedup, কিন্তু memory cost।
• Long context-এ KV cache GB+ — main bottleneck।

Latency breakdown:

• Time to first token: ~১০০ms (prefill phase)।
• Per token after: ~২০-৫০ms।
• ১০০০ token response: ২০-৫০ second।

Cost economics (২০২৪):

• GPT-4 input: $30/1M tokens।
• GPT-4 output: $60/1M tokens।
• Output বেশি costly — sequential decode।
• Caching, speculative decoding — cost কমাতে।

Optimization frontier:

• Speculative decoding: small model draft, large verify। ২-৩x speedup।
• Batching: multiple users together — GPU utilization ৯০%+।
• Quantization: INT8/INT4 weights — memory ২-৪x কম।
• Distillation: small model trained from large।
• Architecture: MoE — same compute, more capacity।

Bangladesh perspective:

• Self-hosting LLM impractical for most।
• API-based usage — cost-aware design।
• Smaller models (7B-13B) — local Bangla LLM possible।
• Edge device — quantized 1-3B model।

মূল উপলব্ধি: Forward pass — DL-এর computational reality। Token-level cost, latency, memory — সবই engineering challenges। Algorithm-level optimization (attention) থেকে hardware-level (GPU, TPU) — সব stack-এ। ChatGPT-এর response speed = forward pass engineering miracle।

Production ML system-এর সবচেয়ে practical engineering trade-off।

Batching-এর সুবিধা:

• GPU memory bandwidth — high throughput।
• Compute efficiency — matrix multiply scaled।
• Cost per inference — ১০-১০০x কমে।

Latency cost:

• Batch fill-এর জন্য wait — first user delayed।
• Single user-এর latency বাড়ে।
• Tail latency unpredictable।

Throughput vs latency curve:

• Batch size 1: low throughput, low latency।
• Batch size 32: 30x throughput, 1.5x latency।
• Batch size 256: 80x throughput, 3x latency।
• Saturation — large batch GPU-saturated।

Dynamic batching:

• Server requests pool — fill batch within time window (e.g., 50ms)।
• Triton, TorchServe, TF Serving — built-in।
• Latency SLA-এর সাথে balance।

Continuous batching (LLM):

• vLLM-এর innovation।
• Different users-এর different sequence length একসাথে।
• Token-level batch — fixed batch size না।
• ২-৪x throughput improvement।

Use case-নির্ভর strategies:

• Real-time speech (voice assistant): small batch, low latency priority।
• Bulk inference (batch prediction): max batch size।
• Web chatbot: dynamic batching, ~৫০ms wait acceptable।
• Real-time CV (autonomous car): single-image, hardware acceleration।

Asynchronous architecture:

• Streaming output — first token দ্রুত, rest stream।
• WebSocket/SSE — incremental delivery।
• User perception of speed > total latency।

Hardware considerations:

• GPU (A100, H100): high batch preferred।
• TPU: static batch size — recompile expensive।
• CPU: small batch acceptable।
• Edge (mobile): single inference, latency-only।

Bangladesh use cases:

• Bangla speech-to-text — streaming, low batch।
• Document OCR — bulk batch।
• Chatbot — dynamic batching।
• Recommendation API — high QPS, max batch।

মূল উপলব্ধি: Batching — DL inference-এর core efficiency lever। কিন্তু blanket "batch বেশি = ভাল" — wrong। Application-নির্ভর tuning। Production ML engineer-এর core skill।

PyTorch-এর সবচেয়ে confusing — কিন্তু critical — detail।

model.eval():

• Model-এর internal flag toggle।
• Affects: Dropout, BatchNorm, certain custom layers।
• Dropout — training-এ random drop, eval-এ identity।
• BatchNorm — training-এ batch statistics, eval-এ running statistics।
• Gradient computation-এ effect নাই।

torch.no_grad():

• Context manager — autograd off।
• Affects: gradient tracking — disabled।
• Memory savings — intermediate activation save করতে হয় না।
• Speed — backward graph build skipped।
• Layer behavior-এ effect নাই।

Difference summary:

• eval() = layer mode change।
• no_grad() = autograd off।
• Independent — দু'টো আলাদা।

Common mistake (১):

• শুধু no_grad() — Dropout এখনো random! Test-এ inconsistent prediction।
• Solution: eval()-ও call করুন।

Common mistake (২):

• শুধু eval() — gradient compute হচ্ছে, memory waste।
• Solution: no_grad()-ও wrap করুন।

Best practice:

model.eval()
with torch.no_grad():
    pred = model(X)

বা even better (PyTorch 1.9+):

model.eval()
with torch.inference_mode():  # stricter, faster
    pred = model(X)

torch.inference_mode():

• no_grad-এর evolved version।
• Tensor mutation also disabled।
• Slightly faster।
• Some restrictions — output tensor in-place modification disallowed।

Training-এর সাথে contrast:

model.train()      # Dropout, BatchNorm-এ train mode
optimizer.zero_grad()
output = model(X)
loss = loss_fn(output, y)
loss.backward()    # gradient compute
optimizer.step()

Validation loop pattern:

for epoch in range(epochs):
    # Training phase
    model.train()
    for X, y in train_loader:
        ...

    # Validation phase
    model.eval()
    with torch.no_grad():
        for X, y in val_loader:
            ...

Subtle pitfall:

• BatchNorm running stats train mode-এ update। eval-এ frozen।
• Test-এ ভুলে train mode — performance vary।

মূল উপলব্ধি: দু'টো mechanism — different concerns. Inference-এ দুটোই দরকার। Production code-এ ভুলে গেলে — hard to debug bug। Always pair them।

PyTorch convention-এর গভীর — যা beginner-দের কাছে cryptic।

"Logits" শব্দটির অর্থ:

• Statistics-এ logit = log-odds, $\log(p/(1-p))$।
• DL-এ — pre-softmax raw output।
• Convention — "scores" বা "raw activations"।
• Range: $(-\infty, +\infty)$।

Logits → probability:

• Binary: $p = \sigma(\text{logit})$।
• Multi-class: $\mathbf{p} = \text{softmax}(\text{logits})$।

কেন softmax model-এ না:

• Numerical stability: raw logits-এ extreme values — softmax overflow। Log-softmax + NLL combine করে stable।
• PyTorch convention: CrossEntropyLoss = log_softmax + NLLLoss internally।
• Direct softmax + log = numerical disaster: $\log(0) = -\infty$।
• LogSumExp trick: CrossEntropy internally uses।

PyTorch's CrossEntropyLoss:

# Input: raw logits (no softmax!)
# Target: class indices
loss_fn = nn.CrossEntropyLoss()
logits = model(X)              # (B, num_classes)
targets = torch.tensor([3, 7, 1, 5])  # class indices
loss = loss_fn(logits, targets)

Beginner mistake:

# WRONG — double softmax!
output = nn.Softmax(dim=1)(logits)
loss = nn.CrossEntropyLoss()(output, targets)  # BUG

Inference-এ softmax কখন apply:

• Probability দরকার — যেমন confidence display।
• Top-k sampling — sampling-এর জন্য probability।
• Calibration analysis।
• আর্গmax-এ — softmax লাগে না (softmax monotonic)।

Binary classification special:

• BCEWithLogitsLoss: sigmoid + BCE combined। Stable।
• BCELoss: sigmoid first (less stable)।
• Convention — WithLogits version preferred।

Why "logits" terminology:

• Multinomial logistic regression থেকে inherited।
• Statistics-এ formal usage।
• "Pre-softmax score" — verbose, "logits" — short।

Practical implications:

• Save model — save logits-producing version।
• Export to ONNX — softmax separate।
• Distillation — KL divergence on logits।
• Interpretation — logits-এর magnitude meaningful (calibration)।

Logit space-এর gain:

• Linear arithmetic — concept editing (control vectors)।
• Knowledge distillation — soft targets।
• Adversarial robustness analysis।

মূল উপলব্ধি: "Logits" — PyTorch idiom। Output-এ softmax না — convention, performance, ও stability-র জন্য। CrossEntropyLoss expects logits — beginner-দের ৩৩% bug এই double-softmax। Convention মেনে চলা = bug-free code।