Multi-head attention

Multi-head vs single-large — fundamental design question। Empirical winner clear, theoretical understanding subtle।

Computational equivalence:

• Single $d$-dim — $O(T^2 d)$।
• $h$ heads $d/h$-dim — $O(h T^2 d/h) = O(T^2 d)$।
• Same compute, different expressivity।

Why split helps:

• Different subspace — different relationship।
• Single attention — single softmax distribution।
• Multi-head — multiple parallel distributions।
• Higher representational capacity।

Subspace specialization:

• Each head — projection to subspace।
• Subspace — particular relationship type capture।
• Linguistic features distributed across heads।
• Multi-task implicitly।

Information bottleneck:

• Single attention softmax — averaging effect।
• Multiple — sharper, focused।
• "Mode collapse" prevent।

Empirical evidence:

• Vaswani (২০১৭) ablation — multi-head significantly better।
• Same parameter, same compute।
• BLEU score significant gain।

Theoretical perspective:

• Average of distributions — sometimes better than single।
• Implicit ensemble।
• Reduce variance।

Per-head capacity:

• $d_k = 64$ — single attention pattern enough capacity।
• $d_k = 8$ — too small।
• $d_k = 256$ — wasteful।
• Sweet spot empirically determined।

Linguistic structure capture:

• Syntax — local dependency।
• Semantics — global relationship।
• Coreference — long-range।
• Position — adjacent।
• Different head — different structure।

Single head equivalence (limit):

• $d_k = d_{model}$, $h = 1$ — single full attention।
• Theoretically can capture multiple pattern।
• Practice — softmax-এর constraints।
• Gradient signal weak across multiple pattern।

Practical hyperparameter:

• $h = 8$ — sweet spot small models।
• $h = 12$ — BERT-base।
• $h = 16, 32$ — larger models।
• $h = 96$ — GPT-3।

Failed modes (single head):

• Diffuse attention — uninformative।
• Single dominant pattern — limited।
• Weaker representation।

Modern perspective:

• Linear attention — head matter less।
• Mixture-of-experts — explicit specialization।
• Cross-attention — multi-head still effective।

Counter-intuitive finding:

• Voita et al. — many heads "redundant" trained model-এ।
• Pruning works — but training-এ all heads contribute।
• Training dynamics ≠ inference।

Bangla NLP context:

• Standard $h = 8$-$12$ ব্যবহার।
• Specific Bangla linguistic feature head specialize possible।
• Verb-subject (Bangla SOV) — specific head।
• Honorific tracking — another head।

মূল উপলব্ধি: Multi-head — same compute, higher expressivity। Different subspace different relationship। Linguistic structure emergent capture। Empirical winner clearly multi-head। Theoretical — implicit ensemble + diverse pattern। Hyperparameter $d_k = 64$-$128$ sweet spot। Modern AI's core ingredient।

BERT attention analysis — interpretability research-এর active area। Clark et al. (২০১৯) "What Does BERT Look At?" landmark paper।

Identified head categories:

• Positional heads: diagonal pattern — adjacent token attention।
• Broad heads: uniform attention — averaging information।
• Specialized heads: specific syntactic role।
• SEP heads: [SEP] token-এ attention sink।

Specific syntactic roles:

• Direct object identification — head 8-10।
• Coreference resolution — head 7।
• Possessive pronoun antecedent — specific head।
• Determiner-noun relation।
• Verb-object pattern।

Layer-wise pattern:

• Lower layers — surface features (position, n-gram)।
• Middle layers — syntactic dependency।
• Higher layers — semantic relationship।
• Hierarchical processing emergent।

Probing methodology:

• Linguistic test — accuracy on specific phenomenon।
• Coreference benchmark।
• Dependency parsing।
• Semantic role labeling।

Bangla-specific predictions:

(১) Verb-final structure:

• Bangla SOV — verb sentence end।
• Special head — verb to subject (long-range)।
• Distance-aware attention।

(২) Honorific tracking:

• "আপনি" vs "তুমি" vs "তুই" — verb-end-এ agreement।
• Specific head pronoun-verb tracking।

(৩) Postposition handling:

• "ঢাকায়" = "ঢাকা + য়" — locative।
• Subword-aware head।
• Compound morpheme tracking।

(৪) Conjunct character:

• "যুক্তাক্ষর" — multi-character formation।
• Local attention important।
• Position head focused।

BanglaBERT analysis:

• Limited public analysis।
• Similar pattern expected (multilingual BERT analyses align)।
• Bangladesh research opportunity।

Visualization tools:

from transformers import BertTokenizer, BertModel
import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased')
model = BertModel.from_pretrained(
    'bert-base-multilingual-cased',
    output_attentions=True)

text = "আমি ঢাকায় থাকি"
inputs = tokenizer(text, return_tensors='pt')
outputs = model(**inputs)
attention = outputs.attentions  # tuple of (B, h, T, T)

# Visualize layer 5, head 8
import matplotlib.pyplot as plt
plt.imshow(attention[5][0, 8].detach())
plt.colorbar()

BertViz library:

• Vig (২০১৯) — interactive attention visualization।
• Multiple view — neuron, layer, head।
• Bangla-friendly (Unicode support)।

Pruning insights:

• Voita — many heads "useless" pruning OK।
• Specific role-এর head essential।
• Redundancy + specialization mix।

Cross-lingual:

• mBERT — multiple language same head।
• Some head universal pattern।
• Some language-specific।
• Bangla — both shared + specific।

Interpretability limit:

• Attention pattern ≠ explanation।
• "Attention is not explanation" (Jain-Wallace ২০১৯)।
• Information flow complex।
• Multi-layer interaction।

Mechanistic interpretability:

• Anthropic recent work — circuit analysis।
• Specific neurons identify।
• Feature visualization।
• Attention head + MLP combined understanding।

Research opportunity:

• Bangla linguistic features which heads capture?
• Compared to English head — differences?
• Cross-lingual transfer — which head transfer?
• Bangladesh academic — open problem।

Practical use:

• Model debugging — pattern check।
• Head pruning — efficiency।
• Custom architecture — head budget allocate।
• Interpretability — model trust।

মূল উপলব্ধি: BERT head — emergent linguistic role। Position, syntax, coreference — specialization clear। Bangla-specific pattern (SOV, honorific, postposition) — research opportunity। Visualization tool available। Pruning shows redundancy + essential mix। Interpretability — DL transparency dimension।

Bangladesh practical scenario — ১২ head BanglaBERT fine-tune, deployment-এ optimization required।

Baseline cost:

• BanglaBERT-base — ১১০M parameter।
• Inference ~১০০ms CPU, ~৫ms GPU।
• Memory ~৪০০MB FP32।
• Production scale-এ expensive।

Strategy 1 — Head pruning:

• Voita method — head importance score।
• ৫০% head prune — accuracy <১% drop।
• Parameter ~25% reduce।
• Inference ~30% faster।

Strategy 2 — Quantization:

• FP32 → INT8 — 4x size reduction।
• Accuracy slight drop।
• Hardware support — most CPU/GPU।
• ৪০০MB → ১০০MB।

Strategy 3 — Knowledge distillation:

• Teacher BanglaBERT-base।
• Student smaller model (DistilBanglaBERT)।
• ৬ layer, ৪০M parameter।
• ~৯৫% accuracy maintain।

Strategy 4 — Layer pruning:

• Top layers often redundant।
• ৬-৮ layer keep।
• Speed-up significant।
• Task-specific evaluation।

Strategy 5 — Mixed-precision:

• FP16 training/inference।
• Memory ½, speed ~2x।
• Modern GPU (V100, A100) native support।
• Negligible accuracy loss।

Strategy 6 — Flash attention:

• Memory $O(T)$ instead $O(T^2)$।
• ৪x longer context same hardware।
• ~২x faster training।
• PyTorch ২.০ built-in।

Strategy 7 — Sliding window:

• Attention window-restricted।
• Long document efficient।
• Quality slight drop।

Implementation pipeline:

# 1. Load pretrained
from transformers import BertModel, BertConfig
model = BertModel.from_pretrained('csebuetnlp/banglabert')

# 2. Knowledge distillation
from transformers import DistilBertConfig, DistilBertModel
student_config = DistilBertConfig(
    n_layers=6, n_heads=12, dim=768)
student = DistilBertModel(student_config)
# Distill train procedure

# 3. Quantization
from torch.quantization import quantize_dynamic
quantized = quantize_dynamic(
    student, {nn.Linear}, dtype=torch.qint8)

# 4. ONNX export
torch.onnx.export(quantized, dummy_input,
                  'banglabert_optimized.onnx')

# 5. Inference engine
import onnxruntime as ort
sess = ort.InferenceSession('banglabert_optimized.onnx')

Combined gains:

• Distillation: 2-3x speed।
• + Quantization: 4x size, 2x speed।
• + Flash attention: 2x speed।
• Combined: 10-20x speedup।

Use case-specific:

Customer service chatbot:

• Real-time — distilled model।
• ৫০ms latency target।
• Mobile/server deployment।

Document classification:

• Batch processing — full BanglaBERT।
• Throughput-optimized।
• GPU server।

Search:

• Cached embedding।
• Approximate nearest neighbor।
• FAISS/Pinecone integration।

Hardware budget:

• Cloud GPU — $500-2000/month।
• CPU server — cheaper, slower।
• Edge device — mobile-friendly model।
• API service vs self-host।

Bangladesh-specific:

• Cost-conscious market।
• Lower compute available।
• Offline/limited connectivity scenario।
• Distillation + quantization essential।

Production monitoring:

• Latency p50, p95, p99।
• Throughput (requests/second)।
• Accuracy A/B test।
• Cost per request।

Continuous improvement:

• New BanglaBERT version।
• Efficient architecture variant।
• User feedback integration।
• Domain-specific fine-tune।

মূল উপলব্ধি: Multi-head efficient deployment — head pruning, quantization, distillation, flash attention combine। 10-20x speedup achievable। Bangladesh — cost-conscious, distillation critical। Production monitoring continuous। Modern AI deployment art + science। Bangla NLP — practical engineering challenge।

৩ paradigm — DL history-এর তিন era। Final synthesis।

Computational properties:

• RNN: $O(T)$ sequential, $O(d^2)$ per step।
• CNN: $O(\log_k T)$ depth, parallel within layer।
• Self-attention: $O(1)$ depth, $O(T^2 d)$ compute।

Path length (i ↔ j):

• RNN: $O(T)$ — long path।
• CNN: $O(\log T)$ — depending depth।
• Attention: $O(1)$ — direct।

Parallelization:

• RNN: poor — sequential।
• CNN: excellent।
• Attention: excellent।

Inductive bias:

• RNN: temporal, recency।
• CNN: locality, translation invariance।
• Attention: minimal — order via positional encoding।

Long-range capture:

• RNN: weak — vanishing gradient।
• CNN: moderate — receptive field expand depth।
• Attention: excellent — direct।

Memory:

• RNN: $O(T)$।
• CNN: $O(T)$।
• Attention: $O(T^2)$।

Translation task:

• RNN-Seq2Seq (২০১৪) — first NMT।
• CNN-Seq2Seq (Gehring ২০১৭) — fast।
• Transformer (২০১৭) — current dominant।

Speech recognition:

• RNN/LSTM (DeepSpeech) — ২০১৪-২০১৮।
• CNN-based (Wave2Vec) — ২০১৯।
• Conformer (CNN+Transformer) — current state-of-art।

Time-series forecasting:

• LSTM/GRU — classical।
• TCN (Temporal Convolutional Network) — strong baseline।
• Informer, Autoformer — Transformer adapted।
• Mixed result — domain-specific।

Image classification:

• CNN dominant ২০১২-২০২০।
• ViT (Vision Transformer) — competitive।
• Hybrid (ConvNeXt) — best both world।

Object detection:

• CNN backbone (Faster R-CNN, YOLO)।
• DETR (Detection Transformer) — competitive।
• Hybrid common।

Generation:

• RNN — char-level (Karpathy classic)।
• Transformer — modern LLM (GPT)।
• Diffusion — image (UNet-based)।

Edge deployment:

• RNN/GRU — small, mobile-friendly।
• CNN — efficient kernels।
• Transformer — distillation/quantization required।

Recent renaissance — RNN comeback:

• Mamba (২০২৪) — state-space model।
• RWKV — RNN modern reformulation।
• $O(n)$ inference — long context efficient।
• Transformer-competitive accuracy।

Hybrid winner pattern:

• Conformer — CNN + attention, speech।
• ConvNeXt — CNN inductive bias + Transformer।
• Mamba-Transformer — Jamba architecture।
• Best both world — common modern trend।

Selection guideline:

Choose RNN/LSTM:

• Edge deployment।
• Online streaming।
• Small data।
• Resource constrained।

Choose CNN:

• Image, audio।
• Local pattern important।
• Translation invariance natural।
• Efficient kernel।

Choose Transformer:

• Large data, large compute।
• Long-range dependency।
• Pretrained model available।
• State-of-art accuracy require।

Bangladesh practical:

• Production NLP — BanglaBERT (Transformer)।
• Mobile NLP — distilled GRU/LSTM।
• Vision — CNN (efficient)।
• Speech — Conformer/hybrid।

Trend prediction:

• Transformer dominate — short to medium term।
• State-space models rise — long context।
• Hybrid common — task-optimized।
• Efficient variant focus।

Universal architecture trend:

• Same Transformer — vision, speech, language।
• Multi-modal model rise।
• Foundation model paradigm।
• Emergent capability।

মূল উপলব্ধি: RNN, CNN, Transformer — তিন era। Each strength specific। Modern dominant Transformer — large data + compute। Hybrid emerging best-both। Bangladesh — pragmatic mix। ML history — architecture innovation cycle। Foundation strong then specific tool select।