GRU — সরল বিকল্প

এটি GRU-এর design genius। Cell state বাদ — কিন্তু gradient highway property সংরক্ষিত। কীভাবে?

GRU-এর update equation:

$h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$

$\partial h_t / \partial h_{t-1} \approx 1 - z_t$ (যখন $\tilde{h}_t$-এ $h_{t-1}$ dependence ignore)।

Highway property:

• $z_t \approx 0$ — $h_t \approx h_{t-1}$, gradient flow $\approx 1$।
• Long-range gradient preserve।
• LSTM-এর forget gate-এর equivalent।

LSTM cell state-এর role:

• "Linear" memory — output gate দিয়ে selective expose।
• Bigger capacity — separate variable।
• Long-range storage explicit।

GRU-এ trade-off:

• Hidden state ও memory একত্রে।
• Output ও memory same — selective output capability কম।
• Capacity LSTM-এর চেয়ে slightly limited।

Mathematical equivalence (paper analysis):

• Greff et al. (২০১৫) — extensive comparison।
• GRU = LSTM minus output gate (approximately)।
• Most task-এ negligible difference।
• Specific task যেখানে output gate matter — LSTM win।

When output gate matters:

• Memory store information necessary later, but not exposed each step।
• Counting tasks (count digits in sequence)।
• Long context task with sparse retrieval।

Empirical observations:

• NLP — GRU very competitive, often winner small data।
• Speech — LSTM slight edge generally।
• Time-series — both comparable।
• Music — LSTM long-term structure better।

Modern perspective:

• Transformer dominate language tasks।
• State-space models (Mamba) — GRU-like simple update।
• RWKV — RNN renaissance, GRU-inspired।

Cell state mathematical preservation:

• LSTM-এ $c_t$ unbounded, additive।
• GRU-এ $h_t$ tanh-bounded।
• LSTM-এ longer-range possible theoretically।
• Practice-এ similar।

Bangladesh deployment context:

• Mobile inference — GRU faster, smaller।
• Battery — fewer ops।
• Memory — important for low-end Android।
• Production — GRU acceptable trade-off।

মূল উপলব্ধি: GRU cell state বাদ — কিন্তু update gate-এর through gradient highway সংরক্ষিত। $(1 - z_t) \odot h_{t-1}$ — same effect। LSTM-এর marginally more capacity — output gate। Practice-এ similar performance। Simple architecture often sufficient। Bangladesh-এ mobile NLP — GRU practical winner।

Convex combination GRU-এর elegant design choice। Two independent gate (LSTM-এর forget + input) থেকে কেন এটা better?

Convex combination property:

• $h_t = (1 - z_t) \cdot h_{t-1} + z_t \cdot \tilde{h}_t$।
• $z_t \in [0, 1]$।
• Output bounded — max norm preserved।
• "Information conservation" — কিছু রাখি, কিছু update।

Two independent gate alternative:

• $h_t = f_t \cdot h_{t-1} + i_t \cdot \tilde{h}_t$ (LSTM-style)।
• $f_t, i_t$ independently $\in [0, 1]$।
• Possible — both ০ (forget all, write nothing → $h_t = 0$)।
• Possible — both ১ (keep all, write all → unbounded growth)।

Coupled gate benefit:

• State magnitude bounded।
• Easier to reason about।
• Single decision — "কতটুকু update"।
• One less hyperparameter।

Independent gate benefit:

• More expressive — separate forget vs add।
• Empirically সামান্য better long-range task।
• LSTM-এর foundation।

Empirical study:

• Cho et al. (২০১৪) — coupled gate equivalent performance।
• Greff et al. (২০১৫) — coupled vs independent — small difference।
• Coupled — fewer parameter, similar accuracy।
• Modern convention — coupled in GRU।

Probabilistic interpretation:

• $z_t$ = "probability of update"।
• $h_t$ = expected value under update decision।
• Soft gate — differentiable।
• Hard gate (binary) — non-differentiable, not used।

Geometric interpretation:

• Hidden state space-এ — convex combination = line segment।
• $h_t$ — $h_{t-1}$ ও $\tilde{h}_t$-এর মধ্যে কোথাও।
• Smooth interpolation।
• Stability promote।

Numerical stability:

• Bounded state — gradient explosion কম likely।
• Normalization-এ আলাদা concern নেই।
• Long sequence training stable।

Modern parallels:

• Highway networks — convex combination input/transform।
• Residual connection — addition without coupling।
• Mixture-of-experts — gate-based routing।

মূল উপলব্ধি: Coupled gate ($z, 1-z$) — single decision "কতটুকু update"। Bounded state, simpler reasoning, fewer params। LSTM-এর independent gate marginally more expressive। Empirically similar। GRU-এর elegance — minimal sufficient mechanism। Architecture design — simplicity vs expressivity balance।

Mobile NLP — Bangladesh-এর জন্য huge opportunity। Bangla input এখনো অনেক users-এর কঠিন। Predictive keyboard — UX revolution।

Constraint analysis:

• Memory: ১০MB — model + vocab embedding।
• Latency: ১০ms — perceived instant।
• Battery: minimal CPU/GPU usage।
• Offline — no internet dependency।

Vocabulary strategy:

• Top ৩০K-৫০K word — coverage ৯০%।
• Embedding ৬৪-D (small)।
• Subword for rare — ৫K subword।
• Total vocab embedding — ~৩MB at FP32।

Architecture:

import torch
import torch.nn as nn

class MobileGRU(nn.Module):
    def __init__(self, vocab=30000, embed=64,
                 hidden=128, num_layers=2):
        super().__init__()
        self.embed = nn.Embedding(vocab, embed)
        self.gru = nn.GRU(embed, hidden,
                          num_layers=num_layers,
                          batch_first=True)
        self.fc = nn.Linear(hidden, vocab)

    def forward(self, x, h=None):
        emb = self.embed(x)
        out, h = self.gru(emb, h)
        logits = self.fc(out[:, -1, :])
        return logits, h

Parameter budget:

• Embedding: ৩০K × ৬৪ = ১.৯২M।
• GRU layer 1: ৩(৬৪+১২৮)১২৮ = ৭৩K।
• GRU layer 2: ৩(১২৮+১২৮)১২৮ = ৯৮K।
• Output: ১২৮ × ৩০K = ৩.৮৪M।
• Total: ~৬M parameter, ২৪MB FP32।

Optimization techniques:

• Quantization: INT8 — size ¼। ৬MB।
• Weight tying: input ও output embedding share। Save ৩.৮M param।
• Smaller vocab: ১৫K — common word coverage ৮৫%।
• Adaptive softmax: common word fast, rare slow।

Final model size:

• Tied weights + INT8 + ১৫K vocab।
• ~১.৫M parameter → ১.৫MB।
• Comfortable under ১০MB budget।

Inference optimization:

• ONNX export — cross-platform।
• TensorFlow Lite — Android native।
• Pruning — sparse weight।
• Flash attention-style memory layout।

Latency breakdown:

• Embedding lookup: <১ms।
• GRU forward: ~৩ms।
• Softmax + top-k: ~৪ms।
• Total: ~৮ms — comfortable।

Training data:

• Bangla Wikipedia, news, social media।
• Crowdsource — actual typed text।
• Privacy concern — federated learning option।
• Personalization — user history fine-tune।

Bangla-specific challenges:

• Code-mix — English + Bangla।
• Multiple keyboard layout (Bijoy, Avro)।
• Phonetic input — "ami" → "আমি"।
• Conjuncts (যুক্তাক্ষর) handling।

Personalization:

• User-specific fine-tune — top layer।
• On-device learning।
• Privacy-preserving।
• Differential privacy if cloud sync।

Beam search / top-k:

def predict_top_k(model, context, k=3):
    with torch.no_grad():
        logits, _ = model(context)
        probs = torch.softmax(logits, dim=-1)
        top_k = torch.topk(probs, k=k, dim=-1)
    return top_k.indices, top_k.values

UX design:

• ৩ suggestion above keyboard।
• Auto-complete on tap।
• Smart correction।
• Emoji integration।

Evaluation metrics:

• Top-1 accuracy — exact match।
• Top-3 accuracy — usable suggestion।
• Keystroke savings — actual UX metric।
• User retention — long-term।

Modern alternative:

• Distilled BERT mobile — accuracy higher, size larger।
• Hybrid — GRU + retrieval।
• On-device transformer (DistilBERT)।
• Trade-off accuracy vs deployment।

Bangladesh market potential:

• ~১৭ কোটি Bangla speaker।
• Smartphone penetration ~৫০%।
• Bangla typing pain point।
• Native solution missing — opportunity।

মূল উপলব্ধি: Mobile next-word — GRU ideal। ১০MB budget, ১০ms latency comfortably achievable। Quantization + weight tying — model compress ১.৫MB। Training Bangla corpus + user personalization। Bangladesh-এ market huge — ১৭ কোটি potential user। Mobile NLP — practical AI deployment-এর shining example।

১০+ বছর empirical study — pattern emerge হয়েছে। Decision framework নিচে।

Major studies:

• Chung et al. (২০১৪): GRU vs LSTM same performance। Music, speech।
• Greff et al. (২০১৫): "LSTM Search Space Odyssey" — variants comparison।
• Jozefowicz et al. (২০১৫): ১০K architecture search। GRU/LSTM near-optimal।
• Bai et al. (২০১৮): TCN > both for many tasks।

Task-specific findings:

NLP — text classification:

• GRU often slight winner (small data)।
• LSTM larger data tie।
• Bi-directional both work well।

Language modeling:

• LSTM slight edge — long context।
• Cell state benefit।
• Difference smaller post-attention।

Machine translation:

• ২০১৪-২০১৭ — LSTM dominate।
• GRU competitive small dataset।
• Now Transformer।

Speech recognition:

• LSTM historical preference।
• Long acoustic sequence — cell state helpful।
• Modern Conformer/Transformer।

Time-series forecast:

• Both comparable।
• Domain expertise > model choice।
• Feature engineering matter more।

Music generation:

• LSTM long-term structure better।
• Counting বার, motif return।
• Cell state explicit memory helpful।

Reinforcement learning:

• GRU faster training — ablate quickly।
• LSTM partial observability complex tasks।

Decision framework:

Step 1 — Compute budget:

• Limited (mobile, embedded) → GRU।
• Generous (server, cloud) → either।

Step 2 — Sequence length:

• Short (<৫০) → GRU sufficient।
• Medium (৫০-৫০০) → either।
• Long (৫০০+) → LSTM advantage।

Step 3 — Data size:

• Small (<১০K) → GRU (less overfit)।
• Medium (১০K-১M) → either।
• Large (১M+) → consider Transformer।

Step 4 — Task complexity:

• Simple (classification) → GRU।
• Complex (counting, retrieval) → LSTM।

Step 5 — Validation:

• Both implement, A/B test।
• Statistical significance check।
• Training time, inference time include।

Modern context:

• NLP — Transformer dominant।
• RNN niches — long sequence, edge, online।
• State-space models — RNN renaissance।
• Mamba — Transformer competitor।

When neither — Transformer:

• Pretrained available (BERT, GPT)।
• Long-range context critical।
• Compute available।
• Parallelization required।

Bangladesh practical recommendation:

• Default — GRU (compute, simplicity)।
• Critical accuracy task — try LSTM।
• Production NLP — BanglaBERT fine-tune।
• Edge deployment — quantized GRU।

Hyperparameter sensitivity:

• LSTM more — forget bias, gate init।
• GRU less — fewer hyperparameters।
• GRU faster experimentation।

Reproducibility:

• Random seed effect significant।
• Multiple runs average।
• ৫% variation common — small difference noise।

মূল উপলব্ধি: GRU vs LSTM — task-dependent, marginal difference often। Decision framework — compute, sequence length, data size, complexity। Default GRU, escalate LSTM if needed। Modern Transformer dominant — কিন্তু RNN renaissance state-space models। Bangladesh — pragmatic GRU। ML-এ "best architecture" নেই — context-dependent। Empirical experimentation crucial।