LSTM — gate-এর সাহায্যে স্মৃতি

এটা LSTM-এর সবচেয়ে গভীর design question। উত্তর — empirical + minimal sufficient information control।

৩ gate-এর rationale:

• Forget — পুরনো clear: পুরনো information irrelevant হলে clear করা দরকার।
• Input — নতুন write: নতুন information selectively লেখা দরকার।
• Output — selective expose: cell state-এর সব information current step-এ দরকার নাও পারে।

২-gate variants:

• GRU (২০১৪, Cho et al.): reset + update। Forget ও input merge — coupled।
• $z_t$ (update) controls কতটুকু change।
• $1 - z_t$ implicitly forget।
• সাধারণত LSTM-এর কাছাকাছি accuracy, fewer parameters।

১-gate hypothesis:

• JZS1, JZS2, JZS3 (Jozefowicz et al., ২০১৫) — RNN architecture search।
• খুব কম gate — performance drop significant।
• ৩ gate sweet spot empirically।

Coupled forget-input (CIFG):

• $f_t = 1 - i_t$ — coupled gate।
• Parameter কম।
• Performance LSTM-এর কাছাকাছি।

Why ৩ specifically work:

• Information theory — ৩ independent control variables (write, erase, expose)।
• Computational graph — ৩ separate gradient flow।
• Empirical tuning ১৯৯৭-এ ৩ gate optimal পেয়েছিল।

Empirical comparison studies:

• Greff et al. (২০১৫) — "LSTM: A Search Space Odyssey"।
• ৮টি variant test করেছিলেন।
• Vanilla LSTM ও peephole comparable।
• Coupled gate fewer params, similar accuracy।
• Output gate removal — significant drop।

Modern perspective:

• Transformer attention — gate-এর alternative formulation।
• Mamba (২০২৪) — selective state update, gate-like।
• "Gating" পুরো DL-এ recurring pattern।

Design principle:

• Information flow control — multiple independent decision।
• Differentiable selection — sigmoid gate।
• Architectural inductive bias।

Bangladesh practical:

• Default LSTM use, GRU compute-constrained-এ।
• Mobile deployment GRU preferred।
• Both production-ready।

মূল উপলব্ধি: ৩ gate — Hochreiter-Schmidhuber-এর empirically tuned। ২-gate (GRU) similar performance। ১-gate insufficient। Information control require multiple independent selection mechanism। Modern architecture-এ gating recurring theme। Design choice empirical + theoretical balance।

Forget gate bias init — LSTM training-এর crucial detail। Default-এ ০ — কিন্তু Jozefowicz et al. (২০১৫) recommend 1.0।

Default ০-init problem:

• Forget gate $f_t = \sigma(W_f \cdot \text{input} + b_f)$।
• Random init-এ initially $f_t \approx \sigma(0) = 0.5$।
• Cell state প্রতি step-এ ৫০% decay।
• Long-range gradient — ০.৫^T → vanish।
• Training শুরুতে — long-range learning impossible।

$b_f = 1.0$ init benefit:

• $f_t \approx \sigma(1) \approx 0.73$।
• Initially "remember" preference।
• Long-range gradient flow শুরু থেকেই possible।
• Network gradually learn কখন forget করতে হবে।

PyTorch implementation:

lstm = nn.LSTM(input_size=10, hidden_size=20)

# Forget gate bias = position 1 of 4 (i, f, g, o)
for name, param in lstm.named_parameters():
    if 'bias' in name:
        n = param.size(0)
        # Each bias = [b_i, b_f, b_g, b_o]
        # Set forget gate bias to 1.0
        param.data[n // 4 : n // 2].fill_(1.0)

Empirical impact:

• Long sequence task — significant improvement।
• Convergence speed faster।
• Final accuracy slight gain।
• Best practice modern LSTM training।

Theoretical justification:

• Bias-এ ১ — gradient flow easier initially।
• Network "forget less" prior।
• Easier to learn "when to forget" than "when to remember"।
• Inductive bias-এ favor remembering।

Other gate biases:

• Input gate: ০ default fine — write decision data-driven।
• Output gate: ০ default fine।
• Forget — special case, init matter।

Larger models:

• Bigger model — gradient pathway more critical।
• Deep stacked LSTM — bias init essential।
• Char-level language model — particularly helpful।

Modern training tricks:

• Layer normalization in LSTM — stabilize।
• Recurrent dropout — variational, not random।
• Weight tying — input/output embedding share।
• Mixed precision training।

Bangla NLP context:

• Bangla sentence — average length ~১৫-২০ token।
• Long document — ১০০-৫০০ token।
• Long-range dependency — agreement, anaphora।
• Forget bias init helpful।

Debugging:

• Gradient norm monitor — initial step ভাল pattern?
• Forget gate value visualize — learning trajectory দেখুন।
• Per-position activation — long-range memory check।

মূল উপলব্ধি: Forget gate bias = ১ — small change, big impact। Default zero init initially "forget half each step" — long-range learning impossible। Bias শুরুতে remembering bias। Best practice modern LSTM। Detail matter — tiny architectural choice training-এ pivotal।

Bi-LSTM vs uni-LSTM — task-এর nature দিয়ে নির্ধারিত। Causality crucial।

Causality definition:

• Causal task — output time $t$-এ শুধু $\le t$ input দেখতে পারে।
• Non-causal — full sequence দেখা যায়।
• Future leak — causal task-এ illegal।

Causal task examples:

• Language modeling — next word predict।
• Time-series forecasting — future predict।
• Real-time speech recognition (streaming)।
• Online text generation।

Non-causal task examples:

• POS tagging — full sentence available।
• NER — context উভয় দিকে।
• Sentiment analysis — পুরো review।
• Translation (encoder side) — full source।
• Audio classification (offline)।

Bi-LSTM benefit non-causal:

• "I went to the bank" — "bank" finance vs river? পরে context লাগে।
• Bangla — "সে __ করে" — "কাজ" vs "লেখা"? উভয় দিক।
• Long-range agreement — verb-subject।
• Disambiguation।

Why uni-LSTM causal:

• Future information — leak হলে test-time impossible।
• Real-time deployment — future নেই।
• Generation — auto-regressive।
• Inference latency — full sequence wait নয়।

Architecture comparison:

• Bi-LSTM — ২ direction, ২ x parameters।
• Hidden ~ ২x size effective।
• Cannot use causal task।
• Concatenation forward + backward।

Workaround for causal + bidirectional context:

• BERT-style: bi-direction encoder, masked LM। Pretrain বড় data, fine-tune downstream।
• Two-pass: first pass generate draft (uni), second pass refine (bi)।
• Self-attention: Transformer — attention mask দিয়ে causal বা bi।

Streaming considerations:

• Bi-LSTM — full sequence wait, latency বাড়ে।
• Uni-LSTM — token-by-token, low latency।
• Hybrid — chunk-based bi-direction।

Bangla NER example:

class BanglaNER(nn.Module):
    def __init__(self, vocab, embed=128, hidden=128, num_tags=9):
        super().__init__()
        self.embed = nn.Embedding(vocab, embed)
        # Bi-LSTM — both context for entity recognition
        self.lstm = nn.LSTM(embed, hidden,
                            bidirectional=True,
                            batch_first=True)
        self.fc = nn.Linear(hidden * 2, num_tags)

    def forward(self, x):
        emb = self.embed(x)
        out, _ = self.lstm(emb)
        # Per-token tag prediction
        return self.fc(out)

Practical decisions:

• Default — task analyse করুন।
• Real-time? Uni।
• Full sequence available? Bi।
• Better accuracy? Bi (যদি permitted)।

মূল উপলব্ধি: Causality task-এর core property। Causal — uni-direction। Non-causal — bi-direction। Bi-LSTM accuracy high কিন্তু applicable subset। Real-time generation forced uni। Modern Transformer attention mask-এ এই control native। Architecture choice always task-driven।

Time-series forecasting — Bangladesh agriculture-এ critical practical problem। চাল, পেঁয়াজ, ডাল — সব price-sensitive।

Data understanding:

• ১০ বছর × ৩৬৫ = ~৩৬৫০ data point।
• Features — price (target), weather, festival, supply।
• Seasonality — yearly cycle, monsoon, harvest।
• External shocks — flood, COVID, war।

Feature engineering:

• Price (lag ১, ৭, ৩০ day)।
• Rolling mean (৭, ৩০ day)।
• Seasonality features (month, day-of-year)।
• Festival flag (ঈদ, পূজা, পৌষমেলা)।
• Weather — temperature, rainfall।
• Macro — fuel price, USD rate।

Architecture:

import torch
import torch.nn as nn

class PriceLSTM(nn.Module):
    def __init__(self, n_features=10, hidden=64,
                 n_layers=2, horizon=30):
        super().__init__()
        self.lstm = nn.LSTM(n_features, hidden,
                            num_layers=n_layers,
                            dropout=0.2,
                            batch_first=True)
        # Multi-step forecast — direct
        self.fc = nn.Linear(hidden, horizon)

    def forward(self, x):
        # x: (batch, lookback, features)
        _, (h, _) = self.lstm(x)
        last = h[-1]
        return self.fc(last)  # (batch, horizon) prices

model = PriceLSTM(n_features=10, hidden=64,
                   n_layers=2, horizon=30)

Sequence preparation:

• Lookback window — ৬০-৯০ day।
• Forecast horizon — ১, ৭, ৩০ day।
• Sliding window — overlapping samples।
• Train/val/test — time-based split (no shuffle)।

Forecasting strategies:

• Direct multi-step: output entire horizon at once। Above implementation।
• Recursive: predict t+1, feed back, predict t+2... । Error compound।
• Encoder-decoder: seq2seq for forecasting। L29 lesson।
• Hybrid often best।

Loss function:

• MSE — standard।
• MAE — outlier robust (price spike)।
• Huber — combine MAE-MSE।
• Quantile loss — uncertainty estimate।
• Custom — economic loss (over vs under prediction)।

Normalization critical:

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
train_norm = scaler.fit_transform(train)
# Always inverse_transform for predictions

Evaluation metrics:

• RMSE — magnitude error।
• MAPE — percentage (interpretable for stakeholder)।
• Direction accuracy — up/down correct?
• Sharpe-like — economic value।

Bangladesh-specific challenges:

• Festival impact — ঈদ-এ price spike।
• Weather extreme — flood disrupt supply।
• Government intervention — TCB price।
• Hoarding — non-stationary।
• Market segmentation — Dhaka vs district।

Model alternatives:

• ARIMA, SARIMA — classical baseline।
• Prophet (Facebook) — easy use।
• LSTM — non-linear pattern।
• Temporal Fusion Transformer — state-of-art।
• Hybrid LSTM + ARIMA।

Production considerations:

• Daily retraining — recent pattern adapt।
• Confidence interval — uncertainty quantify।
• Ensemble multiple model — robust।
• Human-in-the-loop — domain expert review।

Stakeholder integration:

• Ministry of Food — policy decision।
• TCB (Trading Corporation of Bangladesh) — import timing।
• Wholesale market — purchasing decision।
• Farmer — sell timing।

Realistic accuracy:

• Next-day MAPE — ১-৩%।
• ৭-day MAPE — ৫-৮%।
• ৩০-day — ১০-১৫%।
• Direction accuracy — ৬০-৭০%।

Ethical considerations:

• Predictions exposed — speculation potential।
• Public good vs commercial use।
• Government oversight।
• Farmer access।

মূল উপলব্ধি: Bangladesh price prediction — LSTM practical। Feature engineering critical (festival, weather)। Multi-step direct forecasting common। Time-based split মাত্র। Production — daily retrain, uncertainty quantify। Government, business, farmer — multiple stakeholder। Agriculture AI — Bangladesh-এ biggest impact opportunity।