Sequence-to-Sequence

এটি famous practical trick — paper-এ "introducing many short term dependencies" বলে described। বুঝার গভীর insight।

Original setup:

• "আমি বাংলা ভালোবাসি" → encoder reads "আমি, বাংলা, ভালোবাসি"।
• Decoder generates "I, love, Bangla"।
• "আমি" ↔ "I" — এদের মধ্যে gap = source full length + decoder positions।

Reverse trick:

• Encoder reads "ভালোবাসি, বাংলা, আমি"।
• Decoder generates "I, love, Bangla"।
• "আমি" এখন encoder-এর শেষে — context vector-এ fresh।
• "আমি" ↔ "I" — gap minimum।

Why it works:

• Recency bias — context vector-এ শেষ token-এর information dominant।
• Reversed-এ — translation-এর শুরু-এর word encoder-এ শেষ।
• "Many short-term dependencies" — মূল mapping pair-এ gap কম।

Empirical impact:

• BLEU score WMT'14 — ৪.৭ → ৩৫.৬ improvement (with deeper model)।
• Long sentence — particularly helpful।
• Without reverse — long sentence accuracy poor।

Why this trick disappears today:

• Attention mechanism — gap concept-ই irrelevant।
• Decoder প্রতি step-এ source-এর সব position access।
• Reverse implicit-এ achieved।
• Modern Transformer — completely position-flexible।

Related ideas:

• Bidirectional encoder — both forward + backward।
• Attention — direct source position access।
• Copy mechanism — source token directly use।

Theoretical interpretation:

• Vanishing gradient — long path through compression bottleneck।
• Reverse — gradient path shorter for early target tokens।
• Optimization landscape easier।

Modern lesson:

• Architecture limitation — clever data hack overcome।
• "Trick" temporary — better architecture permanent solution।
• Engineering insight valuable in research।

Bangla translation context:

• Bangla SOV (subject-object-verb), English SVO।
• Word order significant difference।
• Reverse trick may help less — verb in Bangla last anyway।
• Modern Transformer handle natively।

মূল উপলব্ধি: Reverse trick — ২০১৪-এ pragmatic engineering। Vanishing gradient + bottleneck-এর symptom। Recency bias exploit। Attention mechanism এর "need" eliminate। Modern Transformer order-agnostic। Lesson — architecture limitation-এর জন্য temporary hack acceptable, কিন্তু ultimately better architecture দরকার।

Beam search NLP generation-এর foundational algorithm। এর nuance বুঝা production-এ critical।

Greedy decoding:

• প্রতি step-এ argmax probability।
• Locally optimal, globally not necessarily।
• Fast — single pass।

Beam search:

• Top-$k$ partial sequences maintain।
• Each step expand each by top-$k$।
• Overall top-$k$ keep।
• Greedy = beam=১।

Quality difference reasons:

• Greedy commit early — backtrack impossible।
• "The dog" vs "The cat" — first word commit affect rest।
• Beam — multiple option keep, recover possible।

Quantitative impact:

• Translation BLEU — beam ৫ vs greedy ~১-৩ point gain।
• Summarization ROUGE — similar।
• Speech recognition WER — beam essential।

$k = \infty$ — exhaustive:

• Theoretically — exact MAP sequence।
• Computationally infeasible — exponential।
• Practical $k$ — ৫-১০ typical।
• $k$ বাড়ালে — diminishing return।

Surprising finding:

• Very large $k$ — quality decrease।
• "Beam search curse" — Murray-Chiang (২০১৮)।
• Reason — model probability mismatch true distribution।
• High-probability sequence often empty/short।

Beam search problems:

• Length bias — short sequence preferred।
• Repetition — same phrase repeat (low-prob loop trap)।
• Diversity low — beam similar each other।

Practical fixes:

• Length normalization: $\log P / |y|^\alpha$, $\alpha = 0.7$।
• Coverage penalty: source attention coverage encourage।
• N-gram blocking: repetition prevent।
• Diverse beam: beam-এর মধ্যে diversity penalize।

Sampling alternatives:

• Top-k sampling: top-k থেকে random। Diversity।
• Nucleus (top-p): cumulative probability $p$ পর্যন্ত। ChatGPT use।
• Temperature: softmax sharpness control।
• Open-ended generation — sampling preferred।

Task-specific guidance:

• Translation — beam search (correctness)।
• Summarization — beam search।
• Story generation — sampling (creativity)।
• Dialog — sampling + temperature।

Computational cost:

• Beam $k$ — memory $k$x, compute $k$x।
• Production — beam ৪-৬ typical।
• Real-time — greedy/beam ২।

Modern context (LLM):

• GPT-style — sampling primarily।
• Beam search — task-specific (translation, code)।
• Constrained generation — beam helpful।

Bangla translation deployment:

• Beam ৫ — quality acceptable।
• Length normalization essential।
• Repetition handling।
• BLEU evaluate।

মূল উপলব্ধি: Beam search — quality boost over greedy, কিন্তু infinite ≠ optimal। Length bias, repetition, diversity issues। Modern sampling (top-p) often preferred open-ended task-এ। Constrained task — beam still relevant। Decoding strategy — task-specific empirical choice।

Bangla translation Bangladesh-এর জন্য huge opportunity। ১৭ কোটি speaker, কিন্তু resource-poor language।

Training data sources:

• BUET Parallel Corpus: ~৩০০K sentence pairs।
• SUST NMT corpus: ~১M পেয়ার।
• Wikipedia Bangla — English aligned: ~২০০K।
• Government documents: Bangladesh Gazette।
• Religious texts: Bible, Quran translations।
• Movie subtitles: OpenSubtitles।

Data quality issues:

• Noisy alignment — sentence-level not always accurate।
• Domain bias — religious, governmental over-represented।
• Modern conversational underrepresented।
• Cleaning crucial।

Data augmentation:

• Back-translation: EN→BN, then BN→EN — synthetic pairs।
• Pivot translation: BN → HI → EN (Hindi pivot)।
• Paraphrase generation: diversity বাড়ান।
• Crowdsource: Bangla speaker translate।

Architecture choice:

• Vanilla Seq2Seq — baseline।
• Seq2Seq + Attention — significant gain।
• Transformer (vanilla) — production quality।
• Pretrained — mBART, IndicBERT fine-tune।
• NLLB-200 — Meta-এর recent multi-lingual।

Tokenization:

• SentencePiece — BPE for both Bangla and English।
• Vocab ~৩২K — balance।
• Bangla যুক্তাক্ষর — special handling।
• Number, URL, name — placeholder।

Training setup:

import torch
from torch.optim.lr_scheduler import LambdaLR

# Adam + warmup + decay
optimizer = torch.optim.Adam(model.parameters(),
                              lr=1e-4)

def lr_schedule(step, warmup=4000, max_lr=1e-4):
    if step < warmup:
        return step / warmup
    return (warmup / step) ** 0.5

scheduler = LambdaLR(optimizer, lr_schedule)

# Label smoothing — overconfidence reduce
criterion = nn.CrossEntropyLoss(label_smoothing=0.1,
                                  ignore_index=PAD)

Evaluation metrics:

• BLEU: n-gram overlap। Standard but flawed।
• ChrF: character F-score। Bangla-এর জন্য ভাল।
• METEOR: semantic alignment।
• BLEURT: learned metric। Best correlation with human।
• Human evaluation: ultimate test।

Bangladesh-specific challenges:

• Code-mix: "আমি office যাচ্ছি" — common। Translation tricky।
• Honorifics: আপনি/তুমি/তুই — English-এ "you" only।
• Named entity: "ঢাকা" → "Dhaka" — copy mechanism helpful।
• Cultural concepts: "মা শা আল্লাহ" — direct translation impossible।
• Idioms: "মাথা গরম" → "angry" (literal "head hot" ভুল)।

Production deployment:

• API service — REST/gRPC।
• Batch translation — government documents।
• Real-time — chat application।
• On-device — privacy-sensitive।
• Caching — frequent translations।

Latency requirements:

• API: ১-২ second per sentence acceptable।
• Real-time chat: <১ second।
• Batch: throughput-optimized।

Cost considerations:

• GPU server — $৫০০-২০০০/মাস depending scale।
• API call vs self-host break-even ~১M call/month।
• Cloud Google/Azure translation expensive long-term।

Current state:

• Google Translate Bangla — reasonable, not perfect।
• Local startup — Pathao, bKash internal use।
• Open-source — IndicTrans2, NLLB available।
• Commercial opportunity — domain-specific (legal, medical)।

Realistic accuracy:

• Vanilla Seq2Seq — BLEU ১৫-২০ (poor)।
• Seq2Seq + Attention — BLEU ২৫-৩০।
• Transformer + sufficient data — BLEU ৩৫-৪০।
• Production threshold — BLEU ৩০+।
• Specialized domain — higher achievable।

মূল উপলব্ধি: Bangla translation — practical impact বিশাল। Vanilla Seq2Seq learning purpose। Production — Transformer + pretrained। Data quality > model complexity। Bangladesh-specific challenge (code-mix, idiom, honorifics)। Continuous improvement requirement। ১৭ কোটি speaker market — opportunity untapped।

Bottleneck — Seq2Seq-এর core limitation। Information theory দিয়ে formally analyze করা যায়।

Information capacity calculation:

• Context vector $\mathbf{c} \in \mathbb{R}^{H}$, FP32।
• Each dim ৩২ bit — practical-এ ২-৪ bit useful (precision)।
• $H = 1024$ — ~২K-৪K bit total।
• Average word — ১৫-২০ bit information (entropy)।
• ~১০০-২০০ word capacity theoretical।

Practical capacity:

• Effective bits — much less।
• Distributed representation — redundant।
• ~৫০ word practical limit।
• Beyond — quality drop sharp।

Empirical observation:

• Cho et al. (২০১৪) — sentence length vs accuracy plot।
• Length ১০ — high accuracy।
• Length ৩০+ — sharp degradation।
• Length ৬০+ — barely usable।

Why information loss:

• Encoder LSTM — forget irrelevant during compression।
• What's "relevant" — task-specific, encoder doesn't know decoder need।
• Some early information lost as later tokens overwrite cell state।
• Single fixed-size representation — fundamentally limited।

Theoretical perspective:

• Mutual information $I(X; \mathbf{c}) \le H(\mathbf{c})$ — bottleneck।
• Long sequence high entropy — impossible to compress losslessly।
• Lossy compression — task-relevant retain, irrelevant discard।
• "Task-relevant" learned — bias inherent।

Workaround attempts:

• Larger context: $H$ increase — diminishing return।
• Multiple context: per-decoder-step আলাদা context।
• Hierarchical: sentence-level, document-level।
• Memory networks: external memory।

Attention solution (preview):

• Decoder প্রতি step-এ encoder hidden states-এর সব access।
• Step-specific weighted combination।
• Effectively variable-size context।
• Bottleneck eliminate।

Information bottleneck principle:

• Tishby et al. — IB principle।
• Optimal compression — task-relevant preserve।
• $\min I(X; T)$ subject to $I(T; Y)$ adequate।
• DL training implicit IB optimization।

Modern alternatives:

• Attention — explicit access।
• Memory networks — external storage।
• Pointer networks — copy mechanism।
• Retrieval-augmented — external knowledge।

Bangla long document:

• News article — ৫০০-১০০০ word।
• Vanilla Seq2Seq — fail।
• Hierarchical encoder — sentence then document।
• Modern Transformer + chunking।

Practical implications:

• Vanilla Seq2Seq — short sentence only।
• Sentence-by-sentence document translation।
• Context lost across sentence — coherence issue।
• Document-level model required।

Lesson for ML design:

• Architectural bottleneck — fundamental limit।
• "More parameter" — not always solution।
• Mechanism design — capacity matter।
• Information flow analysis — design tool।

মূল উপলব্ধি: Bottleneck information-theoretic। Single fixed vector — long sequence impossible। Empirical degradation length ৫০+। Workaround লাগে — attention solution। DL design — information capacity matter। Practical Bangla — short sentence vanilla, long document need attention/Transformer।