পাঠ ১২ · ৪৫-এর মধ্যে · মডিউল ২

Home / AI Courses / Machine Learning / Logistic Regression

Logistic Regression — sigmoid দিয়ে

Logistic regression

৮ মিনিট পড়া মাঝারি · Intermediate NumPy + sklearn

এই পাঠে যা শিখবেন

কেন linear regression classification-এ fail করে
Sigmoid function — গণিত, geometry, AI-তে ভূমিকা
Decision boundary — কোথায় "yes" ও "no"-র মাঝে রেখা
NumPy + sklearn দিয়ে binary classifier বানানো

১ · Linear regression কেন fail করে

ভাবুন আপনি spam detection বানাচ্ছেন। Output: ০ (ham) বা ১ (spam)। Linear regression apply করলে — output range $(-\infty, +\infty)$। যেমন কিছু sample-এ predict $-0.5$, কিছুতে $+1.8$। Probability হিসেবে interpret করা যায় না।

আবার একটি extreme outlier (very obvious spam) থাকলে — line টানতে পারে data পুরো — যেখানে ছোট মান়ুষ মিস্ক্লাসিফাই হয়।

সমস্যা

১) Output bounded চাই — $[0, 1]$ probability।
২) Smooth decision — sharp threshold avoid।
৩) Outlier-resistant — extreme value-এ overreact না।

২ · Sigmoid — সমাধান

SigmoidSigmoid (Logistic) Function$\sigma(z) = 1/(1+e^{-z})$ — যেকোনো real input-কে $[0, 1]$-এ map করে। $z = 0$-তে output $0.5$। AI-তে probability output-এর প্রধান activation। function:

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

বৈশিষ্ট্য:

$z \to +\infty$: $\sigma(z) \to 1$।
$z \to -\infty$: $\sigma(z) \to 0$।
$z = 0$: $\sigma(z) = 0.5$।
S-shaped curve, smooth, differentiable।
Derivative: $\sigma'(z) = \sigma(z)(1 - \sigma(z))$ — elegant।

৩ · Logistic Regression model

Linear part:

$$z = \mathbf{w}^\top \mathbf{x} + b$$

Probability prediction:

$$P(y=1 | \mathbf{x}) = \sigma(z) = \frac{1}{1 + e^{-(\mathbf{w}^\top \mathbf{x} + b)}}$$

Class prediction (threshold ০.৫):

$$\hat{y} = \begin{cases} 1 & \text{if } \sigma(z) \geq 0.5 \\ 0 & \text{otherwise} \end{cases}$$

ভাবুন একজন ডাক্তার রোগী দেখছেন। Symptom-এর সাথে weight যোগ করে একটি "score" — সেটাই $z$। তারপর সেই score → "disease probability" map করেন (০ থেকে ১)। ০.৫-এর বেশি হলে treatment শুরু। এটাই logistic regression।

৪ · Decision boundary

$\sigma(z) = 0.5$ যেখানে — সেটাই decision boundary। অর্থাৎ $z = 0$, যা $\mathbf{w}^\top \mathbf{x} + b = 0$। ২-D-তে এটি একটি straight line; ৩-D-তে plane; n-D-তে hyperplane।

"Logistic" output non-linear (S-shape), কিন্তু decision boundary linear। Non-linear boundary-এর জন্য — feature engineering (polynomial), kernel, বা neural network লাগে।

Linear "score" → sigmoid → probability → class। Binary classification-এর সবচেয়ে standard pipeline।

৫ · NumPy দিয়ে — sigmoid plot

Python · NumPy

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

# কিছু values try
zs = np.array([-5, -2, -1, 0, 1, 2, 5])
print("z      σ(z)")
for z in zs:
    print(f"{z:5.1f}   {sigmoid(z):.4f}")

# Derivative property
print("\nσ'(0) =", sigmoid(0) * (1 - sigmoid(0)))  # 0.25
print("σ'(5) =", sigmoid(5) * (1 - sigmoid(5)))   # close to 0

$z = 0$-তে probability ০.৫। Input symmetric — $\sigma(-z) = 1 - \sigma(z)$। Derivative-এর peak $z=0$-এ; far from origin → near zero (vanishing gradient hint)।

৬ · Logistic regression — শূন্য থেকে

Python · NumPy

import numpy as np

np.random.seed(0)
N = 200
# Two clusters
X1 = np.random.randn(N//2, 2) + np.array([2, 2])
X0 = np.random.randn(N//2, 2) + np.array([-2, -2])
X = np.vstack([X1, X0])
y = np.array([1]*(N//2) + [0]*(N//2))

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

# Initialize
w = np.zeros(2)
b = 0.0
lr = 0.1

for epoch in range(500):
    z = X @ w + b
    p = sigmoid(z)

    # gradient (cross-entropy — পরের পাঠে detail)
    grad_w = X.T @ (p - y) / N
    grad_b = np.mean(p - y)

    w -= lr * grad_w
    b -= lr * grad_b

    if epoch % 100 == 0:
        loss = -np.mean(y*np.log(p+1e-9) + (1-y)*np.log(1-p+1e-9))
        acc = np.mean((p >= 0.5) == y)
        print(f"Epoch {epoch:3d}: loss={loss:.4f}, acc={acc:.3f}")

print(f"\nFinal w = {w}, b = {b:.3f}")

Train চলতে চলতে loss কমছে, accuracy বাড়ছে। ২ cluster well-separated — তাই accuracy ১.০-র কাছে। Decision boundary — diagonal line।

৭ · scikit-learn — এক লাইনে

Python · scikit-learn

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Real dataset — breast cancer (classification)
data = load_breast_cancer()
X, y = data.data, data.target

# Train/test split + scaling
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)
scaler = StandardScaler()
X_tr = scaler.fit_transform(X_tr)
X_te = scaler.transform(X_te)

clf = LogisticRegression(max_iter=1000)
clf.fit(X_tr, y_tr)

print(f"Train accuracy: {clf.score(X_tr, y_tr):.4f}")
print(f"Test accuracy:  {clf.score(X_te, y_te):.4f}")

# Probability output
probs = clf.predict_proba(X_te[:5])
print("\nFirst 5 predictions (P(class 0), P(class 1)):")
print(probs.round(3))

Real data-তে ৯৫%+ accuracy — logistic regression simple কিন্তু powerful। Scaling critical (StandardScaler)। Output probability — থ্রেশহোল্ড adjust করার সুযোগ।

৮ · Probability interpretation — odds & logits

Logistic regression-এর elegant property — coefficient interpretable:

$$\log \frac{p}{1-p} = \mathbf{w}^\top \mathbf{x} + b$$

Left side — log-odds (logit)। অর্থাৎ — মডেল log-odds linear করে। প্রতিটি $w_j$ — feature-এর "log-odds-এর প্রতি unit বৃদ্ধি"।

$w_j = 0.7$ মানে — সেই feature এক unit বাড়লে odds-এ $e^{0.7} \approx 2$ গুণ বাড়ে। Banking risk model-এ এই interpretation regulatory-grade।

৯ · কোথায় Logistic Regression rule করে

Banking: Loan default risk — interpretable + regulatory-friendly।
Medical: Disease screening — calibrated probability।
Marketing: Conversion prediction — feature importance।
Click prediction: Ad CTR — billions of predictions/day।
NLP baseline: Text classification first try।

Logistic regression — production-এ underrated। "AI" কথায় lots of XGBoost/DL — কিন্তু banks, hospitals, governments এখনো logistic regression-ই ব্যবহার করে — কারণ explainable + auditable। Performance critical না হলে সর্বদা baseline।

ভাবনার প্রশ্ন

প্রতিটি প্রশ্ন নিজে কিছুক্ষণ ভাবুন — তারপর "→ উত্তর" চাপুন।

প্র ০১ Logistic regression "linear" কেন বলে — যখন output non-linear (sigmoid)? "Generalized Linear Model" পরিচিতি কী?

এই terminology-গত subtlety — অনেক learner-কে confuse করে।

"Linear" — কোথায়:

Decision boundary linear।
Log-odds linear in features।
Coefficients linear combination।

"Non-linear" — কোথায়:

Output (probability) — sigmoid through linear।
$P(y=1|x)$ vs $x$ — S-curve।
Loss surface — strictly convex but non-linear।

Generalized Linear Model (GLM):

Nelder & Wedderburn ১৯৭২।
Linear regression-এর extension।
Three components: linear predictor, link function, distribution।

GLM components:

Linear predictor: $\eta = \mathbf{w}^\top \mathbf{x} + b$।
Link function $g$: $g(E[y]) = \eta$।
Distribution: Exponential family — Gaussian, Bernoulli, Poisson, Gamma।

Logistic regression — GLM সদস্য:

Distribution: Bernoulli।
Link: logit $g(p) = \log(p/(1-p))$।
Inverse link: sigmoid।

Linear regression — GLM সদস্য:

Distribution: Gaussian।
Link: identity $g(\mu) = \mu$।

অন্য GLMs:

Poisson regression: Count data — link log।
Gamma regression: Continuous positive — link inverse।
Negative binomial: Overdispersed counts।
Multinomial logistic: Multi-class (L14)।

"Linear" — historical:

Statistical history — Fisher, Pearson era।
Matrix algebra approach।
"Linear" in parameters — central abstraction।

Geometric perspective:

Decision boundary — hyperplane (truly linear)।
Probability space — non-linear surface।
"Linear in feature space, non-linear in probability"।

Non-linear extensions:

Polynomial features — non-linear boundary।
Kernel methods (L29) — implicit non-linear mapping।
Neural networks — multiple logistic-like units stacked।

Regulatory implication:

Banking — "linear models" required for explainability।
Logistic regression qualifies।
Coefficient interpretation — business-friendly।

মূল উপলব্ধি: "Linear" model — parameters-এ linearity, not necessarily output-এ। GLM framework — unifying perspective: linear regression, logistic regression, Poisson regression — সব একই concept-এর variations।

প্র ০২ Sigmoid-এর "vanishing gradient" সমস্যা কী? Deep network-এ ReLU কেন replace করেছে?

Sigmoid — historically dominant, এখন rarely used in hidden layers। Story instructive।

Sigmoid derivative:

$\sigma'(z) = \sigma(z)(1 - \sigma(z))$।
Maximum at $z = 0$: $\sigma'(0) = 0.25$।
$|z| > 5$: $\sigma'(z) < 0.01$।
Saturated regions — gradient near zero।

Deep network — chain rule blow:

$L$ layers, each gradient $\leq 0.25$।
Total: $0.25^L$।
$L = 5$: $\sim 10^{-3}$।
$L = 10$: $\sim 10^{-6}$।
Early layers — gradient evaporates।

Symptoms:

Early layers freeze.
Slow training।
Loss plateau।
Pre-2010 — DL stuck at ~5 layers।

ReLU — solution (2010):

$\text{ReLU}(z) = \max(0, z)$।
Derivative: $1$ if $z > 0$, $0$ otherwise।
No saturation for positive inputs।
Gradient flow preserved।

ReLU advantages:

No vanishing: Positive direction।
Sparse activations: Half neurons inactive — efficient।
Computationally cheap: max() vs exp()।
Biological inspiration: Neuron firing threshold।

ReLU disadvantages:

"Dying ReLU" — neuron stuck at zero।
Negative input — no gradient।
Not differentiable at zero।

Variants:

Leaky ReLU: Small slope for negative — escape "dead" state।
PReLU: Learnable negative slope।
ELU: Exponential linear unit।
GELU: Smooth ReLU — Transformer favorite।
SiLU/Swish: $z \cdot \sigma(z)$ — modern choice।

Sigmoid — কোথায় still used:

Output layer (binary): Probability output।
Gating (LSTM, GRU): $[0, 1]$ control।
Attention: কখনো-কখনো (mostly softmax)।
Logistic regression: Single layer — vanishing problem নেই।

Sigmoid alternatives in output:

Logits + BCEWithLogitsLoss — numerical stability।
Direct probability vs। অনেক cases-এ same।

Historical perspective:

1980s-2000s: sigmoid + tanh dominant।
2010-2014: ReLU revolution।
2015+: Specialized variants।
2020+: GELU/SiLU in Transformer।

Theoretical analysis:

Mean field theory — depth-wise variance।
"Edge of chaos" — initialization regime।
Activation choice + initialization জুটি।

মূল উপলব্ধি: Sigmoid logistic regression-এর জন্য perfect, deep network-এ disaster। Activation choice depth-এর সাথে scale করতে হয়। ReLU-এর simplicity revolutionary। আজকের সব DL — কোনো-না-কোনো ReLU variant।

প্র ০৩ Logistic regression-এ output "probability" — সত্যিই কি probability? "Calibration" বলতে কী বোঝায়?

Probability interpretation — sometimes correct, sometimes দেখানো-শুধু। Critical distinction।

Naive interpretation:

Sigmoid output ০.৭ — মডেল ৭০% confident।
সত্যিই কি ৭০% case-এ positive?
মাঝে-মাঝে যেকোনো-অন্যরকম।

Calibration concept:

Model "well-calibrated" — predicted probability ≈ actual frequency।
Predicted ০.৭ — সেই samples-এ ৭০% positive observed।
Reliability diagram — diagonal line।

Logistic regression — well-calibrated:

MLE-trained → probabilities meaningful।
Cross-entropy loss → calibration encouraged।
Linear model + correct features → well-calibrated typically।

Other models — often miscalibrated:

SVM: No native probability।
Random Forest: Probabilities biased toward extremes।
Boosting: Often overconfident।
Neural networks: Notoriously overconfident।

Why miscalibration matters:

Decision making: Optimal threshold depends on calibration।
Risk assessment: Insurance, medical — actual probability needed।
Cost-sensitive: Expected cost calculation requires true probability।
Combining models: Ensemble averaging — calibration matters।

Calibration techniques:

(১) Platt scaling:

Train sigmoid on top of model output।
Two parameters — quick fit।
Originally for SVM।

(২) Isotonic regression:

Non-parametric, monotonic mapping।
More flexible, needs more data।

(৩) Temperature scaling:

NN logits scale by temperature $T$।
Single parameter — neural network calibration।
Guo et al. (2017) — modern standard।

Calibration metrics:

ECE (Expected Calibration Error): Average gap।
Reliability diagram: Visual check।
Brier score: Combined sharpness + calibration।

Production considerations:

Calibrate on validation set, not training।
Recalibrate periodically — drift।
Different calibration per segment (geography, time)।

Domain examples:

Weather forecasting: "70% rain" must mean rain 70% of time।
Medical diagnosis: Risk calibration vital।
Lending: Default probability → loan pricing।

মূল উপলব্ধি: "Probability" output ≠ actual probability automatically। Logistic regression naturally well-calibrated; অনেক modern model — না। Production ML-এ calibration check + adjust regularly।

প্র ০৪ Bangladesh-এ একটি bKash transaction fraud detector — logistic regression কীভাবে design করবেন? কোন challenges?

Real-world fraud detection — classical logistic regression playground।

Problem definition:

Input: transaction details (amount, time, sender, receiver, location)।
Output: fraud probability ($P(\text{fraud})$)।
Decision: threshold-এ block বা review।

Why logistic regression first:

Real-time inference (millisecond)।
Interpretable — regulatory requirement।
Fast training — frequent retraining সম্ভব।
Calibrated probability।

Feature engineering:

(১) Transaction features:

Amount (log-transformed — heavy skew)।
Time of day (cyclical)।
Day of week।
Day of month (salary-day spike)।

(২) User behavior:

Average daily transaction count (last 30 days)।
Average amount per transaction।
Time since last transaction।
Ratio: this amount / average amount।

(৩) Receiver:

New receiver (first time)।
Receiver's history (if known)।
Receiver location।

(৪) Velocity:

Transactions in last 1 hour।
Total amount in last hour।
Distinct receivers in last hour।

(৫) Geographic:

Distance from usual location।
SIM swap detected।
Cross-district transactions।

Challenges:

(১) Class imbalance:

Fraud ≪ ১% of transactions।
Naive accuracy meaningless (99% — predict all normal)।
Solutions: class weights, SMOTE (L43), focal loss।

(২) Concept drift:

Fraud patterns evolve (adversarial)।
Fraudsters learn detection patterns।
Solution: weekly retraining, online learning।

(৩) Feedback loop:

Blocked transactions — true label unknown।
Survivorship bias in training data।
Solution: counterfactual reasoning, propensity scoring।

(৪) Cost asymmetry:

False negative — money lost।
False positive — angry customer।
Solution: cost-sensitive threshold।

(৫) Real-time constraint:

<10ms inference required।
Logistic regression — milliseconds easily।
Feature computation — bottleneck।

Threshold selection:

ROC curve — false positive rate vs। true positive rate।
Cost-weighted optimal point।
Multiple thresholds — auto-block, review, allow।

Production architecture:

Real-time feature store।
Logistic regression first-line।
XGBoost backup for borderline।
Human review for top-1% suspicious।

Compliance considerations:

Bangladesh Bank regulations।
Customer notification requirements।
Audit trail — model decisions explained।
Bias testing — geographic, demographic।

মূল উপলব্ধি: Logistic regression — fraud detection-এর "Swiss Army knife"। Combined with thoughtful features ও business logic — production-grade system। Modern alternatives (XGBoost, NN) — performance edge কিন্তু complexity premium।

অনুশীলন

হিসাব করুন: $w = [1, -1]$, $b = 0$। Input $\mathbf{x} = [2, 1]$।
- $z$ কত? $\sigma(z)$ কত?
- Predicted class কী?
- Input $[1, 2]$-এর জন্য একই calculation।
- $z = 1 \cdot 2 + (-1) \cdot 1 + 0 = 1$, $\sigma(1) \approx 0.731$, class = 1।
- $\mathbf{x}=[1,2]$: $z = 1 - 2 = -1$, $\sigma(-1) \approx 0.269$, class = 0।

NumPy: উপরের NumPy gradient descent code-এ একটি new test point $[1, 1]$ predict করুন।

x_new = np.array([1, 1])
z = x_new @ w + b
p = sigmoid(z)
print(f"P(class=1) = {p:.3f}, predicted = {int(p >= 0.5)}")

চিন্তা: Class imbalance ৯৯:১ — naive accuracy ৯৯%। কোন metric ব্যবহার করবেন? Threshold কীভাবে adjust করবেন?

Precision/recall, F1, ROC-AUC, PR-AUC বেশি informative। Threshold ০.৫-এর জায়গায় ০.১/০.৩ — recall বাড়াতে। Cost-sensitive optimal threshold business-এর FN/FP cost থেকে আসে।