Multiclass softmax
এই পাঠে যা শিখবেন
- Softmax — গণিত, geometry, intuition
- Multinomial logistic regression — multi-class structure
- Numerical stability — log-sum-exp trick
- NumPy implementation, MNIST classifier
১ · ২+ class — কী লাগবে
Logistic regression — ২ class। কিন্তু MNIST-এ ১০ digit, ImageNet-এ ১,০০০ class। প্রতি class-এর জন্য আলাদা score দরকার, এবং সব score একসাথে probability-তে convert হবে — যোগফল ১।
একটি option — "one-vs-rest" (OvR) — প্রতি class-এর জন্য আলাদা binary classifier। কাজ করে কিন্তু probability normalize-এর গ্যারান্টি নেই। Cleaner solution — softmaxSoftmax$K$ real number-কে $K$ probabilities-এ map করে — exponential weighting এবং normalization-এর মাধ্যমে। Multi-class classification-এর default activation।।
২ · Softmax formula
Input: scores ("logits") $\mathbf{z} = (z_1, z_2, \ldots, z_K)$।
Output: probabilities $\mathbf{p} = (p_1, \ldots, p_K)$।
$$p_i = \text{softmax}(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}$$
বৈশিষ্ট্য:
- সব $p_i \in (0, 1)$।
- $\sum p_i = 1$ — valid probability distribution।
- "Soft" version of $\arg\max$ — largest $z$ → largest $p$।
- $K = 2$: sigmoid recovered।
- Differentiable — gradient descent friendly।
৩ · Multinomial logistic regression
Each class $c$-এর নিজস্ব weight vector $\mathbf{w}_c$ ও bias $b_c$:
$$z_c = \mathbf{w}_c^\top \mathbf{x} + b_c$$
$$P(y = c | \mathbf{x}) = \frac{e^{z_c}}{\sum_{k=1}^{K} e^{z_k}}$$
Total parameters: $K \times (n+1)$ — $K$ classes, $n$ features + bias each।
৪ · Categorical cross-entropy
Ground truth $y$ — one-hot encoded ($y_c = 1$ for true class, ০ elsewhere)। Loss:
$$L = -\sum_{c=1}^{K} y_c \log p_c = -\log p_{\text{true class}}$$
অর্থাৎ — true class-এর predicted probability log নেওয়া (negative)। True class-এ confidence বাড়ালে loss কমে।
$$\frac{\partial L}{\partial z_c} = p_c - y_c$$
Sigmoid + BCE-র মতোই simple — predicted probability minus true label। সব multi-class neural network এই formula-তে শেখে।
৫ · NumPy দিয়ে — softmax
import numpy as np
def softmax(z):
# numerical stability — subtract max
z_shifted = z - np.max(z, axis=-1, keepdims=True)
exp_z = np.exp(z_shifted)
return exp_z / np.sum(exp_z, axis=-1, keepdims=True)
# Single example
z = np.array([2.0, 1.0, -0.5])
p = softmax(z)
print("logits:", z)
print("probs :", p.round(4))
print("sum :", p.sum())
# Batch
Z = np.array([[2.0, 1.0, -0.5],
[-1.0, 3.0, 0.5],
[0.0, 0.0, 0.0]])
P = softmax(Z)
print("\nBatch probs:")
print(P.round(3))৬ · Numerical stability — log-sum-exp
Naive: $\log \sum e^{z_i}$ — large $z$-এ overflow। Trick:
$$\log \sum_i e^{z_i} = z_{\max} + \log \sum_i e^{z_i - z_{\max}}$$
All exponents now $\leq 0$ — bounded। PyTorch/TF-এর log_softmax এই trick use করে — তাই always preferred।
৭ · Multi-class classifier — শূন্য থেকে
import numpy as np
np.random.seed(0)
N, n, K = 300, 4, 3 # 300 samples, 4 features, 3 classes
# Synthetic data
true_W = np.random.randn(n, K)
X = np.random.randn(N, n)
y = np.argmax(X @ true_W + np.random.randn(N, K) * 0.5, axis=1)
# One-hot encode
Y = np.eye(K)[y]
def softmax(z):
z = z - z.max(axis=-1, keepdims=True)
e = np.exp(z)
return e / e.sum(axis=-1, keepdims=True)
# Initialize
W = np.zeros((n, K))
b = np.zeros(K)
lr = 0.1
for epoch in range(500):
z = X @ W + b
p = softmax(z)
# gradient
grad_W = X.T @ (p - Y) / N
grad_b = (p - Y).mean(axis=0)
W -= lr * grad_W
b -= lr * grad_b
if epoch % 100 == 0:
loss = -np.mean(np.log(p[np.arange(N), y] + 1e-9))
acc = np.mean(np.argmax(p, axis=1) == y)
print(f"Epoch {epoch:3d}: loss={loss:.4f}, acc={acc:.3f}")৮ · sklearn দিয়ে — Iris classification
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
X, y = load_iris(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
scaler = StandardScaler()
X_tr = scaler.fit_transform(X_tr)
X_te = scaler.transform(X_te)
# multinomial — softmax-based
clf = LogisticRegression(multi_class='multinomial', max_iter=200)
clf.fit(X_tr, y_tr)
print(f"Test accuracy: {clf.score(X_te, y_te):.4f}")
# Probabilities
p = clf.predict_proba(X_te[:3])
print("\nFirst 3 predictions (P over 3 classes):")
print(p.round(3))multinomial = softmax based।
৯ · Softmax কোথায় ব্যবহৃত
- Image classification: CIFAR, ImageNet — last layer সর্বদা softmax।
- Language models: Vocabulary (৫০K-১০০K tokens) — next token probability।
- Translation: Target language vocab।
- Speech: Phoneme/word distribution।
- Attention: Transformer-এর attention weights — softmax-normalized।
- RL: Policy network — action distribution।
ভাবনার প্রশ্ন
প্রতিটি প্রশ্ন নিজে কিছুক্ষণ ভাবুন — তারপর "→ উত্তর" চাপুন।
প্র ০১ "Softmax temperature" কী? Sampling-এ কেন গুরুত্বপূর্ণ — GPT-এর "creativity" controlled হয় কীভাবে?
Temperature — Generative AI-এর সবচেয়ে practical hyperparameter।
Modified softmax:
$$p_i = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}$$
- $T$ — temperature parameter।
- $T = 1$ — standard softmax।
- $T \to 0$ — argmax (one-hot)।
- $T \to \infty$ — uniform distribution।
Effect on distribution:
(১) Low $T$ (০.১-০.৫):
- Sharp distribution।
- High-confidence picks dominate।
- "Greedy" sampling।
- Predictable, repetitive।
(২) High $T$ (১.৫-২.০):
- Flatter distribution।
- Lower-probability options possible।
- "Creative" sampling।
- Risk: incoherent।
GPT/LLM use:
- $T = 0$: deterministic (chat default sometimes)।
- $T = 0.7$: balanced creativity (common)।
- $T = 1.5$+: experimental, exploratory।
- "Temperature" — user-facing knob।
Use cases:
- Code generation: Low $T$ (০.২) — correctness first।
- Creative writing: High $T$ (০.৮-১.০) — diversity।
- Q&A: Low $T$ (০) — factual precision।
- Brainstorming: Higher $T$ — multiple options।
Combined with other techniques:
(১) Top-k sampling:
- Top-k tokens রেখে others zero।
- Renormalize।
- Temperature + top-k together।
(২) Top-p (nucleus) sampling:
- Cumulative probability $p$ পর্যন্ত tokens।
- Adaptive vocabulary size।
- Holtzman et al. (2019)।
(৩) Repetition penalty:
- Recently used tokens penalized।
- Boredom prevention।
Knowledge distillation:
- Teacher temperature $T \approx 4$।
- Soft targets — informative।
- Student learns relative probabilities।
- Hinton et al. (2015) seminal।
Calibration connection:
- Temperature scaling — post-hoc calibration।
- Validation set-এ optimal $T$।
- Single parameter — easy fit।
Statistical mechanics origin:
- Boltzmann distribution: $p \propto e^{-E/kT}$।
- Physical temperature ↔ ML temperature।
- "Energy" = negative logit।
Empirical guidance:
- Start with $T = 1.0$।
- Greedy needed: $T = 0.1$ or hard argmax।
- Diversity needed: $T = 0.8-1.2$।
- Tune via human evaluation।
মূল উপলব্ধি: Temperature — sampling distribution control। Production GenAI-এ default user knob। Theoretical foundation deep, practical impact massive।
প্র ০২ "Softmax bottleneck" সমস্যা কী? Mixture of Softmaxes কেন helpful — GPT-এর scale পর্যন্ত matter করে?
Softmax — universal কিন্তু theoretical limitation আছে।
Bottleneck observation:
- Yang et al. (2018) — "Breaking the Softmax Bottleneck"।
- Hidden state $h$ → logits → softmax — rank limited।
- $h$-এর dimension $d$ — matrix factorization rank।
- Real language distributions — high-rank।
Mathematical setup:
- $P_{\theta}(\cdot | c) = \text{softmax}(W h_c)$।
- $h_c \in \mathbb{R}^d$ context embedding।
- $W \in \mathbb{R}^{V \times d}$ — output matrix।
- Logit matrix rank $\leq d$।
Real distribution rank:
- Word probabilities — context-dependent।
- Different contexts — different "modes"।
- Language genuine high-rank — possibly $V$।
- $d \ll V$ → expressivity limit।
Mixture of Softmaxes (MoS):
$$P(\cdot | c) = \sum_{k=1}^{K} \pi_k \cdot \text{softmax}(W_k h_c)$$
- $K$ separate softmax components।
- Mixture weights $\pi_k$।
- Effective rank — $K \cdot d$।
Empirical results:
- Penn Treebank — perplexity drop।
- WikiText-2 — improvement।
- Modest gains, more compute।
Modern perspective:
- GPT-4 / Llama — hidden dim ~১২৮০০।
- Vocabulary ~১০০K-৫০০K।
- $d > \log V$ — bottleneck weaker।
- "Scale solves it"।
Alternative solutions:
(১) Increase hidden dim:
- Direct। Computational cost।
- Modern LLMs — extreme dimensions।
(২) Adaptive softmax:
- Frequent/rare words — different capacity।
- Compute-efficient।
- Grave et al. (2017)।
(৩) Continuous output:
- Predict embedding directly।
- No discrete softmax।
- Research direction।
(৪) Hierarchical softmax:
- Tree structure — log-vocab compute।
- Mikolov et al. (2013)।
- Quality compromise।
Practical implications:
- Small models — bottleneck noticeable।
- Large models — empirically minor।
- Research vs production gap।
Recent developments:
- Sparse attention — different bottleneck।
- State space models (Mamba) — alternative architecture।
- Continuous embeddings — generative।
Connection to LM scaling:
- Chinchilla scaling law — params, data, compute।
- Hidden dim ~ $O(\sqrt{params})$।
- Bottleneck disappears at scale।
মূল উপলব্ধি: Softmax theoretical limitation existed; engineering solved it via scale। MoS — academic interest, less production। Modern LLM — bottleneck practically nonexistent। ML research — sometimes problem definitively solved by compute।
প্র ০৩ K = 1000+ classes-এ softmax compute-expensive। Negative sampling, hierarchical softmax — কীভাবে কাজ করে? Trade-offs কী?
Large vocabulary classification — computational frontier।
সমস্যা:
- $K$ classes — $K$ exponentials, $K$ divisions।
- Backward — $K$ gradients propagate।
- Memory — $K \times d$ output matrix।
- Vocab ১M+: GPU melts।
Approaches:
(১) Negative sampling (Mikolov 2013):
- True class + $k$ random "negative" samples।
- Binary classification each।
- Skip-gram word2vec foundation।
- Compute: $O(k+1)$ vs $O(K)$।
Negative sampling formula:
$$\log \sigma(v_o^\top h) + \sum_{i=1}^{k} \mathbb{E}_{w_i \sim P_n} [\log \sigma(-v_{w_i}^\top h)]$$
- $P_n$ — noise distribution (typically unigram^0.75)।
- $k$ = ৫-২০ for small data।
(২) Hierarchical softmax:
- Vocabulary — binary tree (Huffman tree often)।
- Path from root to leaf = decisions।
- $\log_2 K$ binary classifications।
- Compute: $O(\log K)$।
Tree construction:
- Frequent words — short paths।
- Rare words — long paths।
- Huffman coding intuition।
(৩) Adaptive softmax (Grave 2017):
- Frequency-based clusters।
- High-frequency — full capacity।
- Low-frequency — reduced dim, two-stage।
- Modern competitive choice।
(৪) Sampled softmax:
- Training — random subset of vocabulary।
- Importance sampling correction।
- Inference — full softmax।
- TensorFlow built-in।
(৫) NCE (Noise Contrastive Estimation):
- Generalization of negative sampling।
- Theoretical guarantees।
- Asymptotic equivalence to softmax।
Trade-offs comparison:
Negative sampling:
- + Simple, fast।
- + Empirically excellent (word2vec)।
- − No proper probability distribution।
- − Inference unclear।
Hierarchical softmax:
- + Proper probabilities।
- + $O(\log K)$ inference।
- − Tree structure choice matters।
- − Implementation complex।
Adaptive softmax:
- + Strong empirical performance।
- + Frequency-aware।
- − Cluster definition tuning।
- − More hyperparameters।
Modern LLM era:
- GPT — full softmax (vocab ~৫০K manageable)।
- Tied embeddings — input/output share weights।
- Mixed precision (fp16) — memory reduce।
- Compute scale — direct softmax viable।
Recommendation systems:
- Item vocabulary millions।
- Two-tower retrieval — top-k candidates।
- Re-rank with full softmax on candidates।
মূল উপলব্ধি: Large $K$ — softmax computational frontier। Algorithm choice depends scale, accuracy, training/inference asymmetry। Modern LLM — engineering solved via tied embeddings + scale।
প্র ০৪ "Softmax confidence" — overconfident bias। Calibration কীভাবে fix? Temperature scaling production-এ deploy কীভাবে?
Modern NN — accurate কিন্তু overconfident। Production deployment-এ critical।
Overconfidence problem:
- Modern CNN — ৭০% accuracy কিন্তু ৯০% confidence claim।
- Guo et al. (2017) — landmark study।
- Calibration error 10-30% common।
- Reliability undermined।
Causes:
- Cross-entropy minimization — push to ০ or ১।
- BatchNorm interactions।
- Architecture/depth effects।
- Capacity exceeds data complexity।
Temperature scaling solution:
- Validation set-এ $T$ optimize।
- $p_i = \text{softmax}(z_i / T)$।
- Scale-only — accuracy unchanged।
- Single parameter — overfitting risk minimal।
Pseudocode:
- Train model normally।
- Freeze model, get logits on val set।
- Optimize $T$ minimizing NLL on val।
- Deploy with fixed $T$।
Empirical $T$:
- Modern CNN — $T \approx 1.5-2.5$।
- Higher than ১ — softer probabilities।
- Reliability improvement dramatic।
Calibration metrics:
(১) ECE (Expected Calibration Error):
- Bin predictions by confidence।
- Per-bin: $|\text{accuracy} - \text{confidence}|$।
- Weighted average।
(২) Maximum Calibration Error (MCE):
- Worst bin error।
- Worst-case guarantee।
(৩) Reliability diagram:
- Visual diagnostic।
- Confidence vs accuracy plot।
- Diagonal = perfect calibration।
Beyond temperature:
(১) Vector scaling:
- Per-class temperature।
- $K$ parameters।
- More flexibility, more risk।
(২) Matrix scaling:
- Linear transform of logits।
- $K^2$ parameters।
- Overfit-prone।
(৩) Dirichlet calibration:
- Distribution-aware।
- Strong theoretical foundation।
(৪) Histogram binning:
- Non-parametric।
- Classical, simple।
- Discrete output।
(৫) Ensemble methods:
- Multiple model average।
- Naturally calibrated।
- Compute expensive।
Production deployment:
(১) Calibration setup:
- Held-out calibration set।
- Different from training/validation।
- Representative of deployment distribution।
(২) Monitoring:
- ECE tracked over time।
- Drift detection — recalibrate।
- A/B test with/without calibration।
(৩) Decision interface:
- Calibrated probability — threshold setting।
- Cost-sensitive optimal threshold।
- Confidence intervals possible।
Domain examples:
- Medical: Risk scores must calibrate।
- Autonomous: Action decisions probability-based।
- Financial: Risk pricing accurate probabilities।
Training-time approaches:
- Label smoothing — implicit calibration help।
- Mixup — extreme version।
- Focal loss — sometimes worsens calibration।
- MC dropout — uncertainty estimation।
Limitations:
- Distribution shift — calibration degrades।
- Out-of-distribution samples — over-/underconfident।
- Active research area।
মূল উপলব্ধি: Modern softmax outputs — probabilities-এর illusion। Calibration cheap insurance। Production ML — discriminative + calibrated। Trust depends on it।
অনুশীলন
-
হিসাব করুন: Logits $z = [1, 2, 3]$।
- Softmax কত?
- True class = 2 হলে CE loss কত?
- $e^1, e^2, e^3 \approx 2.72, 7.39, 20.09$। Sum $\approx 30.2$।
- Probs $\approx [0.09, 0.24, 0.67]$।
- True class = 2 → loss $= -\log 0.67 \approx 0.40$।
-
NumPy: Logits $[10, 1000, -3]$ — direct softmax fail করবে কেন? Stable version দেখান।
import numpy as np z = np.array([10, 1000, -3]) # Direct: np.exp(1000) overflow # Stable: z_shift = z - z.max() e = np.exp(z_shift) print(e / e.sum()) # ~[0., 1., 0.] -
চিন্তা: Bangla handwritten digit recognition (MNIST-Bangla) — softmax classifier কীভাবে design করবেন? Class বেশি (চিহ্ন + matra)?
১০-৫০ classes possible। Image (28×28) → flatten → linear → softmax(K)। Cross-entropy loss। Class imbalance handle। Confusion matrix-এ দেখুন কোন class confused।
আরও পড়ুন
- পাঠ ১৫ · Ridge ও Lasso পরবর্তী পাঠ Regularization — overfitting prevent।
- পাঠ ১৩ · Cross-entropy আগের পাঠ Multi-class loss-এর foundation।
- সব AI Courses ABCL TECH Python, ML, DL, NLP, CV, GenAI, RL, MLOps।