Attention mechanism

Soft vs hard — attention design-এর fundamental dichotomy। Differentiability gradient-based learning-এর key।

Soft attention:

• $\alpha_i \in [0, 1]$ continuous — softmax output।
• Weighted combination — সব hidden states contribute (varying degrees)।
• Differentiable — gradient backprop through।
• Standard practice modern DL।

Hard attention:

• একটি specific position select — discrete decision।
• $\alpha$ — one-hot vector।
• Non-differentiable — REINFORCE/Gumbel-softmax লাগে।
• Memory efficient — শুধু selected position store।

Soft-এর benefit:

• End-to-end gradient flow — easy training।
• Multiple position-এর partial focus capture।
• Smooth optimization landscape।
• Interpretable weight visualization।

Soft-এর drawback:

• সব position compute — $O(T)$ memory ও compute।
• Long sequence — expensive।
• Diffuse attention — sometimes unfocused।

Hard-এর benefit:

• Sparse computation — speed gain।
• Focused decision — interpretable।
• Inference time — faster।

Hard-এর challenges:

• REINFORCE — high variance gradient।
• Training unstable।
• Performance generally lower।

Hybrid approaches:

• Sparsemax: soft কিন্তু sparse (some weights exact zero)।
• Top-k attention: soft within top-k।
• Local attention: window-restricted।
• Routing transformer: sparse via clustering।

Modern context:

• Vanilla Transformer — soft, full O(n²)।
• Long sequence — sparse attention popular।
• Linear attention — efficient approximation।
• Flash attention — same compute, optimized memory access।

Theoretical perspective:

• Soft = expected hard — differentiable surrogate।
• Both converge to similar pattern।
• Soft training, hard inference — efficient deployment।

Image captioning case:

• Xu et al. (২০১৫) — both compared।
• Soft — better metrics, smoother attention।
• Hard — sharper visualization, harder train।

Vision Transformer:

• Patches-এ soft attention।
• Image-এ visual concept emerge।
• Interpretability via attention map।

মূল উপলব্ধি: Soft attention — differentiability key, end-to-end gradient flow। Hard discrete, REINFORCE হার্ডে। Modern DL — soft default। Sparse approximation efficiency-এ। Soft-হার্ড equivalence theoretical। Practical — soft sufficient most cases। Differentiability — DL-এর fundamental enabler।

দু'টি foundational attention variant। Choice subtle — both correct, context-dependent।

Bahdanau (২০১৪) — additive:

• $\text{score} = v^\top \tanh(W_1 s + W_2 h)$।
• Learned MLP — flexible।
• Different dimension $s, h$ accommodate (separate $W_1, W_2$)।
• Slow — multiple matmul, tanh nonlinearity।

Luong (২০১৫) — dot-product:

• $\text{score} = s^\top h$।
• সরল — single matmul।
• Same dimension required।
• GPU efficient — batched matmul।

Computational comparison:

• Bahdanau — $O(THd)$ per step, $T$ encoder positions, $d$ hidden।
• Luong — $O(THd)$ same big-O, smaller constant।
• Practice — Luong ৩-৫x faster।

Empirical comparison:

• Translation tasks — similar accuracy।
• Bahdanau marginal edge small data।
• Luong scales better।
• Modern Transformer — scaled dot-product (Luong descendant)।

Why dot-product won:

• GPU friendly — matmul highly optimized।
• Scales — large model possible।
• Self-attention compatible।
• Simple — fewer parameter।

Scaling factor (Transformer):

• $\text{score} = s^\top h / \sqrt{d}$।
• Large $d$ — dot-product variance grow।
• Softmax saturate — gradient vanish।
• $\sqrt{d}$ scale — variance ১, gradient flow।

Different formulations:

• General: $s^\top W h$ — learnable matrix।
• Concat: $v^\top \tanh(W [s; h])$ — Bahdanau variant।
• Location: $W s$ — pure decoder-based।
• Content + location: hybrid।

Modern attention (Transformer):

• $\text{Attn}(Q, K, V) = \text{softmax}(QK^\top / \sqrt{d}) V$।
• Scaled dot-product।
• Generalize — Q, K, V separate projection।
• Multi-head — diverse attention।

Implementation detail:

• Bahdanau — separate W_1, W_2 layer।
• Luong — single matmul (or none)।
• Memory — Bahdanau intermediate tensor large।

Specialized usage:

• Speech recognition — Bahdanau (ESPNet)।
• Translation — Luong/Transformer।
• Image — pixel attention (CBAM) — variants।
• Time-series — additive helpful।

When Bahdanau preferred:

• Encoder-decoder dimension different।
• Small model, small data।
• Interpretability key।
• Custom architecture।

When Luong preferred:

• Large model — speed critical।
• Modern Transformer-compatible।
• GPU efficiency।
• Standard NLP।

Bangla translation:

• Default Luong — speed, modern stack।
• Bahdanau — research/teaching।
• Production — Transformer attention।

মূল উপলব্ধি: Bahdanau (additive) vs Luong (dot-product) — design choice। Empirically similar accuracy, Luong faster। Modern Transformer scaled dot-product — Luong heir। GPU efficiency winning factor। Implementation detail matter at scale। Both correct, context-driven choice।

Attention visualization — model debugging-এর powerful tool। Interpretability gives actionable insight।

Healthy attention pattern:

• Diagonal-like — monotonic alignment।
• Bangla SOV vs English SVO — slight off-diagonal expected।
• Sharp focus — confidence high।
• Coverage — সব source token visited।

Pathological pattern 1 — Diagonal absent:

• Random attention — model not learning alignment।
• Cause — insufficient training, poor init, learning rate issue।
• Fix — train more, check optimizer।

Pattern 2 — Always last token:

• Attention "collapse" to single position।
• Decoder-encoder bottleneck similar issue।
• Fix — temperature in softmax, attention dropout।

Pattern 3 — Repetition:

• Same source token attended multiple decoder steps।
• Output repetition issue।
• Fix — coverage penalty, n-gram blocking decoding।

Pattern 4 — Skipping:

• Source token never attended।
• Translation incomplete।
• Fix — coverage encouragement loss।

Bangla-specific patterns:

• Verb position: Bangla last → English mid। Off-diagonal at end।
• Postpositions: "ঢাকায়" → "in Dhaka" — split attention।
• Honorifics: "তিনি"/"সে" — attention captures formality।
• Compound verbs: "করে দিয়েছিল" — multi-token attention।

Visualization implementation:

import matplotlib.pyplot as plt

def plot_attention(src_tokens, tgt_tokens, alpha):
    # alpha: (len_tgt, len_src)
    fig, ax = plt.subplots(figsize=(10, 8))
    ax.imshow(alpha.cpu(), cmap='viridis', aspect='auto')
    ax.set_xticks(range(len(src_tokens)))
    ax.set_xticklabels(src_tokens, rotation=45,
                       fontfamily='Anek Bangla')
    ax.set_yticks(range(len(tgt_tokens)))
    ax.set_yticklabels(tgt_tokens)
    ax.set_xlabel('Source (Bangla)')
    ax.set_ylabel('Target (English)')
    plt.colorbar(ax.images[0])
    plt.tight_layout()
    plt.show()

Diagnostic checklist:

• Multiple example visualize — sample diversity।
• Length variation — short ও long both।
• Domain variation — formal ও colloquial।
• Error case especially — what attention says?

Quantitative metrics:

• Entropy: attention sharpness measure।
• Coverage: source token attended at least once?
• Monotonicity: diagonal-like score।
• Alignment error rate (AER): human-aligned vs predicted।

Connection to translation quality:

• Sharp attention — high confidence, often correct।
• Diffuse — uncertainty, often error।
• Pathological pattern strong correlation translation quality।

Improvement strategies:

• Coverage loss: coverage vector penalize repeated attention।
• Attention regularization: entropy bonus or sparsity।
• Multi-head: different head capture different alignment।
• Cross-attention layer-wise: Transformer multiple level।

Bangla-English specific:

• Word order flip — verb attention key insight।
• Subject-honorific agreement — attention captures?
• Conjunct character — multi-position attention।
• Code-mix English token — attention pattern?

Production debugging:

• Failure case attention save।
• Pattern cluster — common error mode identify।
• Targeted data augmentation।
• A/B test improvement।

Modern context:

• Transformer multi-head — interpretability complex।
• Different head different role।
• Layer-wise pattern emerge।
• Attention rollout — aggregate visualization।

মূল উপলব্ধি: Attention visualization — model debug-এর powerful tool। Pattern healthy vs pathological distinguish। Bangla-English alignment — verb, postposition specific। Coverage, monotonicity, entropy quantitative metric। Production-এ continuous monitoring। Interpretability — DL-এর rare gift, fully exploit।

Attention scaling — modern AI-এর core bottleneck। ChatGPT-এর "context length" এই issue-এর সরাসরি ফল।

Standard attention complexity:

• $Q \in \mathbb{R}^{n \times d}$, $K \in \mathbb{R}^{n \times d}$।
• $QK^\top \in \mathbb{R}^{n \times n}$ — $O(n^2 d)$ compute, $O(n^2)$ memory।
• $n = 1000$ — ১M attention scores।
• $n = 100,000$ — ১০B scores। Infeasible।

Sparse attention:

• Local window: $k$-nearest position। $O(nk)$।
• Strided: every $s$-th position।
• Sparse Transformer (২০১৯): mixed pattern।
• Longformer (২০২০): sliding + global token।
• BigBird: random + window + global।

Linear attention:

• Reformulate $\text{softmax}(QK^\top)V \to \phi(Q)(\phi(K)^\top V)$।
• $\phi$ — kernel feature map।
• $O(n d^2)$ — linear in $n$।
• Performer (২০২০), Linear Transformer (২০২০)।
• Approximation — small accuracy loss।

Flash attention (২০২২):

• $O(n^2)$ compute same, কিন্তু $O(n)$ memory।
• Tile-based — GPU memory hierarchy exploit।
• Tri Dao paper — exact attention।
• Training ২-৪x faster, longer context।

State-space models (Mamba):

• Recurrent formulation — $O(n)$ inference।
• Selective mechanism — content-aware।
• Transformer-competitive accuracy।
• RWKV — similar idea, RNN renaissance।

Linformer:

• $K, V$ project to lower dimension।
• $O(nk)$ where $k \ll n$।
• Approximation — most info preserve।

Practical context lengths:

• BERT (২০১৮) — ৫১২ token।
• GPT-3 (২০২০) — ২,০৪৮।
• GPT-4 (২০২৩) — ৩২,০০০ → ১২৮,০০০।
• Claude (২০২৪) — ২০০,০০০।
• Gemini ১.৫ — ১,০০০,০০০।

Trade-off matrix:

• Standard: exact, $O(n^2)$ memory।
• Sparse: approximate, $O(nk)$।
• Linear: approximate, $O(nd)$।
• Flash: exact, $O(n)$ memory।
• Mamba: different paradigm, $O(n)$।

When use which:

• Short context (<১K) — standard fine।
• Medium (১K-৩২K) — Flash attention।
• Long (৩২K+) — sparse + Flash।
• Very long (১M+) — Mamba/sparse।

Memory hierarchy:

• HBM — slow, large।
• SRAM — fast, small।
• Flash — SRAM fully exploit।
• Memory bandwidth bottleneck — compute-bound to memory-bound।

Bangladesh deployment context:

• Document analysis — long context need।
• Bangla legal document — ১০-৫০ page।
• Standard attention — fail।
• Sparse/Flash — production necessary।

Implementation considerations:

• HuggingFace — flash attention support।
• PyTorch ২.০ — built-in optimization।
• Mamba implementation — selective state space।
• Hardware-aware — A100, H100 differ।

Future direction:

• Hybrid — Transformer + state space।
• Mixture of experts — conditional compute।
• Hierarchical — document structure exploit।
• Retrieval-augmented — external memory।

Cost economics:

• $O(n^2)$ — context $২ \times$ → cost $৪ \times$।
• $O(n)$ — context $২ \times$ → cost $২ \times$।
• API pricing reflect this — long context expensive।

মূল উপলব্ধি: Attention $O(n^2)$ — modern AI's central scaling challenge। Sparse, linear, Flash, Mamba — diverse approaches। ১M context length now possible। Bangladesh long document need this। Hardware-software co-design key। Future — hybrid architecture। AI's frontier — context length scaling।