পাঠ ১৩ · ৪৫-এর মধ্যে · মডিউল ২

Home / AI Courses / Machine Learning / Cross-entropy loss

Cross-entropy loss

৬ মিনিট পড়া মাঝারি · Intermediate NumPy কোডসহ

এই পাঠে যা শিখবেন

MSE কেন classification-এ ভাল না — gradient pathology
Cross-entropy — formula, intuition, geometry
Information theory perspective — entropy ও KL
NumPy দিয়ে loss compute, sigmoid-এর সাথে nice gradient

১ · MSE কেন classification-এ ভাল না

Classification-এ true label $y \in \{0, 1\}$, prediction $p \in [0, 1]$। MSE: $(y - p)^2$।

সমস্যা ১ — wrong gradient scale: মডেল confidently wrong (যেমন $y=1$ কিন্তু $p=0.01$) — gradient তবু sigmoid-এর saturated region-এ — small।

সমস্যা ২ — non-convex with sigmoid: Sigmoid + MSE — loss surface non-convex, multiple local minima।

সমস্যা ৩ — দুর্বল penalty signal: Wrong-by-a-lot prediction-কে sufficient punishment নেই।

২ · Cross-entropy — সমাধান

Cross-entropyCross-EntropyInformation theory থেকে আসা — দু'টি probability distribution-এর "মিল" পরিমাপ। Classification-এ true label-এর সাথে predicted probability-র gap। binary form:

$$\text{BCE}(y, p) = -[y \log p + (1-y) \log(1-p)]$$

আবিস্কার:

$y = 1$: loss $= -\log p$। $p \to 1$: loss $\to 0$. $p \to 0$: loss $\to \infty$।
$y = 0$: loss $= -\log(1-p)$। উল্টো pattern।
"Confidently wrong" — infinite penalty।
"Confidently right" — zero penalty।
Symmetric — class-balanced।

কেন এত elegant

Sigmoid-এর সাথে combined gradient simplifies — $\nabla L = (p - y) \cdot \mathbf{x}$। Sigmoid-এর saturation problem নেই, gradient সবসময় meaningful। Convex। Maximum Likelihood interpretation।

৩ · Maximum Likelihood interpretation

Bernoulli model: $P(y|x) = p^y (1-p)^{1-y}$।
Log-likelihood: $\log P = y \log p + (1-y) \log(1-p)$।
Negative log-likelihood (NLL) = Cross-entropy।

অর্থাৎ — cross-entropy minimize = data-এর likelihood maximize। MLE-র direct ML translation।

৪ · Information theory perspective

Entropy: True distribution-এর uncertainty। $H(y) = -\sum y_i \log y_i$।
Cross-entropy: True $y$-এর সাপেক্ষে predicted $p$-এর "average code length"। $H(y, p) = -\sum y_i \log p_i$।
KL divergence: $D_{KL}(y \| p) = H(y, p) - H(y)$ — extra cost।

ভাবুন আপনি একটি code design করছেন বার্তা পাঠানোর জন্য। True distribution জানলে — optimal code (entropy)। ভুল distribution ধরে নিলে — extra bits লাগবে (cross-entropy = entropy + KL)। ML-এ predicted $p$ ভুল হলে — KL-এর penalty।

Cross-entropy — confident wrong prediction-কে aggressively punish।

৫ · NumPy দিয়ে — compute

Python · NumPy

import numpy as np

def bce(y, p, eps=1e-9):
    """Binary cross-entropy."""
    p = np.clip(p, eps, 1 - eps)  # numerical stability
    return -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))

# Various scenarios
y = np.array([1, 0, 1, 0])
preds = {
    "Perfect":     [0.99, 0.01, 0.99, 0.01],
    "Good":        [0.8, 0.2, 0.7, 0.3],
    "Random":      [0.5, 0.5, 0.5, 0.5],
    "Bad":         [0.2, 0.8, 0.3, 0.7],
    "Catastrophic":[0.01, 0.99, 0.01, 0.99],
}

for name, p in preds.items():
    print(f"{name:14s} → BCE = {bce(y, np.array(p)):.4f}")

Perfect: ~0। Random (০.৫): ~0.69 = $\log 2$। Catastrophic: ~4.6+ — exponentially worse। সঠিক direction-এ confident হওয়া reward, ভুল direction-এ penalty।

৬ · Sigmoid + cross-entropy — magic gradient

Logistic regression-এর loss এক sample-এ:

$$L = -[y \log \sigma(z) + (1-y) \log(1 - \sigma(z))]$$

$z$-এর সাপেক্ষে gradient — derive করলে দেখবেন (chain rule):

$$\frac{\partial L}{\partial z} = \sigma(z) - y = p - y$$

Beautifully simple — error itself। সেই কারণেই logistic regression GD-এ implementation neat:

$$\nabla_{\mathbf{w}} L = (p - y) \cdot \mathbf{x}, \quad \nabla_b L = (p - y)$$

৭ · sklearn-এ ব্যবহার

Python · scikit-learn

import numpy as np
from sklearn.metrics import log_loss

y_true = np.array([0, 1, 1, 0, 1])
y_pred_good = np.array([0.1, 0.9, 0.85, 0.2, 0.95])
y_pred_bad  = np.array([0.6, 0.4, 0.5, 0.6, 0.4])

print(f"Good predictions log-loss: {log_loss(y_true, y_pred_good):.4f}")
print(f"Bad predictions log-loss:  {log_loss(y_true, y_pred_bad):.4f}")

# class_weight বদলানোর effect
print(f"Class-weighted: {log_loss(y_true, y_pred_good, sample_weight=[1, 5, 5, 1, 5]):.4f}")

log_loss = cross-entropy। সব major library default loss এটাই। Sample weights যোগ করে imbalanced classes handle।

৮ · Common pitfalls

$\log 0$ → $-\infty$: Always clip $p \in [\epsilon, 1-\epsilon]$।
NaN propagation: Numerical underflow দ্রুত spread।
BCEWithLogitsLoss preferred: PyTorch/TF-এ — sigmoid + BCE combined — numerically stable।
Imbalanced classes: Naive CE — minority class উপেক্ষা। Solution: class weights, focal loss।
Label noise: CE label-noise-এ sensitive। Smoothing (0.9 instead of 1.0) — robustness।

৯ · Beyond binary — preview

Multi-class generalization — categorical cross-entropy:

$$L = -\sum_{c=1}^{C} y_c \log p_c$$

$y$ — one-hot encoded ground truth। $p$ — softmax output (L14)। Concept identical, dimension expanded।

Cross-entropy AI-র প্রতিটি classification model-এ। GPT — token prediction-এ cross-entropy। ImageNet — image class-এ cross-entropy। Spam detection-এ cross-entropy। আজকের সব generative model — pretty much এই loss-এ trained।

ভাবনার প্রশ্ন

প্রতিটি প্রশ্ন নিজে কিছুক্ষণ ভাবুন — তারপর "→ উত্তর" চাপুন।

প্র ০১ "Information" ও "entropy" — Shannon-এর ১৯৪৮-এর কাজ। ML-এ এই concept-গুলো কীভাবে এলো? KL divergence-এর তিনটি interpretation কী?

Information theory ↔ ML connection — surprisingly deep ও beautiful।

Shannon (১৯৪৮):

"A Mathematical Theory of Communication"।
Entropy = uncertainty পরিমাপ।
$H(X) = -\sum p(x) \log p(x)$।
"Bits" = information content।

Entropy intuition:

Coin toss (fair) — $H = 1$ bit।
Biased coin (90% heads) — $H = 0.47$ bit।
Certainty (always heads) — $H = 0$।
"Surprise" → information।

Cross-entropy:

True distribution $p$, predicted $q$।
$H(p, q) = -\sum p(x) \log q(x)$।
$q = p$ হলে → $H(p)$।
$q \neq p$ হলে → $H(p) + D_{KL}(p \| q)$।

KL divergence — তিন interpretation:

(১) Coding cost:

Optimal code for $p$ — $H(p)$ bits average।
Suboptimal $q$-এ encode — $H(p, q)$ bits।
Extra cost = $D_{KL}$।
"Wrong assumption-এর penalty"।

(২) Likelihood ratio:

$D_{KL}(p\|q) = E_p[\log(p/q)]$।
"$p$ vs $q$ — which is more likely on average?"
Hypothesis testing foundation।

(৩) Bayesian update:

Prior $q$, posterior $p$।
$D_{KL}(p\|q)$ = "information gained"।
Mutual information based on KL।

ML connections:

Maximum likelihood: CE minimize = $D_{KL}$ minimize।
Bayesian inference: Variational methods use KL।
VAE: KL regularizer in loss।
Mutual information: Self-supervised learning objectives।

Properties:

$D_{KL} \geq 0$, $= 0$ iff $p = q$।
Asymmetric: $D_{KL}(p\|q) \neq D_{KL}(q\|p)$।
Not a true metric (no triangle inequality)।
Jensen-Shannon — symmetric variant।

Forward vs reverse KL:

Forward $D_{KL}(p\|q)$: "Mean-seeking" — $q$ covers all of $p$।
Reverse $D_{KL}(q\|p)$: "Mode-seeking" — $q$ concentrates।
VAE uses reverse, MLE uses forward।

Modern ML usage:

Knowledge distillation — student matches teacher distribution।
Policy gradient (RL) — KL constraint।
GAN — JS divergence (related)।
Information bottleneck — KL-based regularization।

Bayesian deep learning:

Weight uncertainty → posterior over weights।
Variational inference — KL between approximations।
Evidence Lower Bound (ELBO) — KL-based।

মূল উপলব্ধি: Information theory — ML-এর দ্বিতীয় ভাষা। Loss functions, regularization, generative models — সব এর উপরে গড়া। MLE-cross-entropy-KL trinity central।

প্র ০২ "Focal loss" কী — কেন imbalanced classification-এ cross-entropy-র চেয়ে ভাল? Object detection-এ কেন crucial?

Focal loss — Lin et al. (2017) RetinaNet paper — object detection-এর landmark contribution।

Standard CE-র সমস্যা imbalance-এ:

Class imbalance ১:১০০০ (যেমন object detection — মাত্র কিছু pixel object)।
Easy negative samples massive in number।
Loss dominated by easy examples।
Hard examples drowned out।

Focal loss formula:

$$\text{FL}(p_t) = -(1 - p_t)^\gamma \log p_t$$

$p_t = p$ if $y = 1$, $p_t = 1-p$ if $y = 0$।
$\gamma \geq 0$ — focusing parameter।
$\gamma = 0$ — standard CE।
$\gamma = 2$ — typical।

কীভাবে কাজ করে:

Easy example ($p_t \to 1$): $(1-p_t)^\gamma \to 0$ — loss → 0।
Hard example ($p_t$ small): $(1-p_t)^\gamma \approx 1$ — full loss।
Down-weights easy, focuses hard।

Concrete example:

$p_t = 0.9$, $\gamma = 2$: weight $= 0.01$ (১০০× smaller)।
$p_t = 0.5$, $\gamma = 2$: weight $= 0.25$।
$p_t = 0.1$, $\gamma = 2$: weight $= 0.81$।

Object detection context:

Image-এ কয়েক হাজার anchor boxes।
মাত্র কিছু — object।
Background → easy negative।
Foreground class confusion → hard।
Focal loss — hard examples-এ training focus।

Alternatives:

(১) Class weights:

Minority class higher weight।
Simple, broadly applicable।
Doesn't distinguish hard vs easy।

(২) Hard negative mining:

Top-k worst predictions select।
Heuristic, two-stage training।

(৩) Focal loss:

Soft, automatic hard mining।
Single-stage — one loss।
Gradient flow-friendly।

(৪) Dice loss:

Segmentation-এ popular।
Class overlap optimize।
Class-imbalance robust।

Hyperparameter $\gamma$:

$\gamma = 0$: equivalent to CE।
$\gamma = 0.5, 1, 2, 5$ — common settings।
Higher $\gamma$ — more focus on hard, but unstable training।
$\alpha$-balanced focal loss — class weight added।

Beyond detection:

Imbalanced classification — broader use।
Medical imaging (lesion detection)।
Fraud detection।
Rare event prediction।

Implementation:

PyTorch — manual implementation common।
torchvision — official version।
Numerical stability — log-space computation।

Empirical findings:

RetinaNet — first single-stage detector matching two-stage।
RetinaNet AP improved ~৫% over CE।
YOLO-এর later versions incorporate focal loss।

মূল উপলব্ধি: Cross-entropy excellent default, but task-specific variants superior। Imbalance, hard examples-এ focal loss surgical। Loss design — ML-এর underrated art।

প্র ০৩ "Label smoothing" কী? কেন overconfidence-এ cure? Big language model training-এ ব্যবহৃত হয় কেন?

Label smoothing — subtle technique, large impact। Modern DL essential trick।

Standard CE — hard targets:

Ground truth: one-hot vector $[0, 0, 1, 0]$।
Model push: $p_3 \to 1$, others $\to 0$।
Confidence unbounded encouraged।

সমস্যা:

Model overconfident — calibration bad।
Logits become extreme।
Generalization can suffer।
Sensitive to label noise।

Label smoothing fix:

Replace $1$ with $1 - \epsilon$।
Replace $0$ with $\epsilon / (K-1)$।
$\epsilon$ — smoothing parameter (0.1 typical)।
Soft target: $[\epsilon/3, \epsilon/3, 1-\epsilon, \epsilon/3]$।

Effect:

Model can't push probabilities to 0 or 1।
Gradient persists even at high accuracy।
Implicit regularization।
Calibration improves।

Information theory:

Smoothed label = mixture of one-hot and uniform।
Higher target entropy।
Penalty for low-entropy predictions।

BLEU score connection:

Original NMT paper (Vaswani 2017) used label smoothing।
Slightly worse perplexity, better BLEU।
Translation quality > raw probability।

LLM training:

GPT family — label smoothing common।
Reduces confident wrong tokens।
Improves sampling quality।
Better calibration → safer outputs।

Image classification:

ImageNet training — standard trick।
+0.5-1% accuracy improvement common।
Better calibration metrics।

Mathematical view:

$L = (1-\epsilon) \cdot \text{CE}(y, p) + \epsilon \cdot \text{CE}(\text{uniform}, p)$।
Second term — entropy regularization।
Encourages high-entropy predictions।

When NOT to use:

Noisy labels — already implicit smoothing।
Small classes (binary, sometimes okay)।
When extreme confidence needed (decision systems)।

Variants:

Knowledge distillation: Teacher model's soft labels।
Mixup: Sample interpolation — extreme smoothing।
Temperature scaling: Post-hoc calibration।

Theoretical analysis:

Müller et al. (2019) — confidence penalty interpretation।
Logit space — pulls toward zero।
Margin shrink, but well-calibrated।

Practical guidance:

$\epsilon = 0.1$ default for image classification।
$\epsilon = 0.1-0.2$ for NMT।
Tune via validation।
Monitor calibration metrics।

মূল উপলব্ধি: Label smoothing — simple yet powerful। Hard labels অতিরিক্ত optimistic; soft labels truthful। Modern training pipeline-এ default tool।

প্র ০৪ Sigmoid + MSE = non-convex loss surface। কেন? Cross-entropy + sigmoid = convex। গাণিতিকভাবে দেখান।

Loss surface convexity — optimization theory-র heart।

Convexity matter:

Convex → unique global minimum।
Gradient descent guaranteed converge।
No local minima trap।

MSE + sigmoid analysis:

$L = (y - \sigma(z))^2$।
$z = \mathbf{w}^\top \mathbf{x} + b$।
$\sigma$ — non-linear।
$L$ — non-convex in $\mathbf{w}$।

Hessian check:

Second derivative শুধু positive না সব $z$-এ।
Inflection points present।
Multiple local minima possible।

Visual intuition:

$y = 1$, very negative $z$: $\sigma(z) \approx 0$, error large।
Gradient (chain rule): $-2(y - \sigma(z)) \sigma'(z)$।
$\sigma'(z) \to 0$ when $z \to -\infty$।
Gradient vanishes — flat region — local "minimum" looking।

CE + sigmoid analysis:

$L = -[y \log \sigma(z) + (1-y) \log(1 - \sigma(z))]$।
$\partial L / \partial z = \sigma(z) - y$।
$\partial^2 L / \partial z^2 = \sigma(z)(1 - \sigma(z)) > 0$।
Strictly convex in $z$।

Convexity in $\mathbf{w}$:

$z$ linear in $\mathbf{w}$ ($z = \mathbf{w}^\top \mathbf{x}$)।
Convex composed with linear → convex।
Sum of convex (over samples) → convex।
Therefore $L(\mathbf{w})$ convex।

Beautiful gradient:

$\nabla_{\mathbf{w}} L = (\sigma(z) - y) \mathbf{x}$।
Pure error × input।
No saturation problem।
Always meaningful learning signal।

Why MSE + sigmoid fails:

$\nabla_{\mathbf{w}} L_{\text{MSE}} = -2(y - \sigma(z)) \sigma'(z) \mathbf{x}$।
Extra $\sigma'(z)$ factor।
Far from boundary — vanishes।
Confidently wrong → no learning।

"MLE perspective":

CE = NLL of Bernoulli model।
MLE asymptotic guarantees apply।
MSE — Gaussian assumption — wrong for binary।

Generalized linear models:

Each distribution → "natural" loss।
Bernoulli → CE।
Gaussian → MSE।
Poisson → Poisson NLL।
Always convex with canonical link।

Empirical evidence:

MSE + sigmoid — slow convergence, often plateau।
CE + sigmoid — fast, clean convergence।
Standard practice across all libraries।

NN extension:

Single-layer convex result holds।
Multi-layer — non-convex regardless of loss।
But CE still helps gradient flow।
"Vanishing gradient" partially mitigated।

মূল উপলব্ধি: Loss-activation pairing matters। Sigmoid + CE = mathematical match made in heaven। Convexity, gradient cleanliness, MLE interpretation — সব align। MSE + sigmoid = pedagogically tempting trap। GLM framework — সবগুলো একসাথে clarity।

অনুশীলন

হিসাব করুন: $y = 1$, $p = 0.9$ — BCE কত? $y = 0$, $p = 0.9$ — কত?
- $y = 1, p = 0.9$: $-\log 0.9 \approx 0.105$।
- $y = 0, p = 0.9$: $-\log 0.1 \approx 2.303$।
- একই $p$, opposite $y$ — দ্বিতীয়টা ২২× costlier।
NumPy: $y = [1, 0, 1]$, $p = [0.99, 0.5, 0.01]$ — BCE কত? কোন sample সবচেয়ে কষ্টদায়ক?
```
import numpy as np
y = np.array([1, 0, 1])
p = np.array([0.99, 0.5, 0.01])
bce_per_sample = -(y*np.log(p) + (1-y)*np.log(1-p))
print(bce_per_sample)  # ~[0.01, 0.69, 4.6]
```
তৃতীয় sample — confidently wrong — cost dominate।
চিন্তা: Spam detector-এ false positive (ham → spam) costlier হলে — loss কীভাবে adjust করবেন?

Class-weighted CE — ham class-এর weight বাড়ান। Threshold higher (০.৭ থেকে ০.৯)। অথবা cost-sensitive learning — explicit cost matrix।