পাঠ ১৮ · ৪৫-এর মধ্যে · মডিউল ২

Home / AI Courses / Machine Learning / Naive Bayes

Naive Bayes — Bayes-এর প্রয়োগ

Naive Bayes classifier

৭ মিনিট পড়া মাঝারি · Intermediate NumPy + sklearn

এই পাঠে যা শিখবেন

Bayes' theorem — recap ও ML interpretation
"Naive" assumption — কী, কেন কাজ করে
Gaussian, Multinomial, Bernoulli NB — কখন কোনটি
Bangla spam classifier বানানো — sklearn-এ

১ · Bayes' theorem recap

$$P(y | \mathbf{x}) = \frac{P(\mathbf{x} | y) \cdot P(y)}{P(\mathbf{x})}$$

$P(y)$ — prior (class-এর base rate)।
$P(\mathbf{x} | y)$ — likelihood (class given features generate করার probability)।
$P(\mathbf{x})$ — evidence (normalize)।
$P(y | \mathbf{x})$ — posterior (যা চাই — given features, predict class)।

Classification — $\arg\max_y P(y | \mathbf{x})$ → $\arg\max_y P(\mathbf{x} | y) P(y)$ ($P(\mathbf{x})$ class-independent)।

২ · "Naive" assumption

$P(\mathbf{x} | y)$ compute কঠিন — features-এর joint distribution। Naive BayesNaive Bayesসব features class-given conditionally independent ধরে নেওয়া। অসংখ্য কেস-এ ভুল assumption — তবু classifier accurate। simplification — features class-given conditionally independent:

$$P(\mathbf{x} | y) = \prod_i P(x_i | y)$$

এটা "naive" — features সাধারণত correlated। কিন্তু:

Computation drastically reduce।
Few parameters — overfitting prone না।
Empirically — surprisingly accurate।
Calibration imperfect, কিন্তু ranking correct।

ML in one line

$$\hat{y} = \arg\max_y P(y) \prod_i P(x_i | y)$$

৩ · তিন variant

$P(x_i | y)$ — কী distribution? Variant-অনুসারে:

Gaussian NB: $x_i$ continuous, normally distributed per class। $P(x_i | y) = \mathcal{N}(\mu_y, \sigma_y^2)$।
Multinomial NB: $x_i$ count (e.g., word frequency)। Text classification dominant।
Bernoulli NB: $x_i$ binary (word present/absent)। Document classification।

৪ · Gaussian NB — formula

প্রতি class $y$, প্রতি feature $i$ — $\mu_{y,i}, \sigma_{y,i}^2$ MLE estimate (sample mean ও variance)।

Likelihood:

$$P(x_i | y) = \frac{1}{\sqrt{2\pi \sigma_{y,i}^2}} \exp\left(-\frac{(x_i - \mu_{y,i})^2}{2\sigma_{y,i}^2}\right)$$

Numerical underflow এড়াতে — log probability:

$$\log P(y | \mathbf{x}) \propto \log P(y) + \sum_i \log P(x_i | y)$$

৫ · Multinomial NB — text-এর জন্য

Document — bag-of-words। Each word $w$, class $c$:

$$P(w | c) = \frac{N_{w,c} + \alpha}{\sum_{w'} N_{w',c} + \alpha V}$$

$N_{w,c}$ — class $c$-এ word $w$-এর count। $V$ — vocabulary size। $\alpha$ — Laplace smoothing (সাধারণত ১) — unseen words probability ০ avoid।

Training — class-wise word frequency। Predict — multiply probabilities, take argmax।

৬ · NumPy দিয়ে — Gaussian NB

Python · NumPy

import numpy as np

class GaussianNB:
    def fit(self, X, y):
        self.classes = np.unique(y)
        self.priors = {}
        self.means = {}
        self.vars = {}
        for c in self.classes:
            Xc = X[y == c]
            self.priors[c] = len(Xc) / len(X)
            self.means[c] = Xc.mean(axis=0)
            self.vars[c] = Xc.var(axis=0) + 1e-9  # smooth
        return self

    def _log_prob(self, x, c):
        # log P(class) + sum log P(x_i | class)
        lp = np.log(self.priors[c])
        m, v = self.means[c], self.vars[c]
        lp += -0.5 * np.sum(np.log(2*np.pi*v) + (x - m)**2 / v)
        return lp

    def predict(self, X):
        out = []
        for x in X:
            scores = [self._log_prob(x, c) for c in self.classes]
            out.append(self.classes[np.argmax(scores)])
        return np.array(out)

# Test
np.random.seed(0)
X = np.vstack([np.random.randn(50, 2),
               np.random.randn(50, 2) + np.array([3, 3])])
y = np.array([0]*50 + [1]*50)

clf = GaussianNB().fit(X, y)
acc = np.mean(clf.predict(X) == y)
print(f"Train accuracy: {acc:.4f}")

Two well-separated Gaussians — accuracy near ১.০। Compute fast — শুধু means + variances। No iterative training।

৭ · sklearn দিয়ে — text classification

Python · scikit-learn

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split

# Bangla mini-corpus (toy)
texts = [
    "আপনি লটারি জিতেছেন এখনই ক্লিক করুন",
    "ফ্রি অফার সীমিত সময়ের জন্য",
    "টাকা ইনভেস্ট করে দ্বিগুণ পান",
    "মিটিং আগামীকাল সকাল ১০টায়",
    "আপনার অর্ডার সফলভাবে delivered",
    "ধন্যবাদ আজকের কথোপকথনের জন্য",
    "জরুরী একাউন্ট verify করুন",
    "প্রজেক্ট রিপোর্ট পাঠানো হলো",
    "প্রাইজ মানি claim করুন এই লিংকে",
    "পরশু lunch করব?",
]
labels = [1, 1, 1, 0, 0, 0, 1, 0, 1, 0]  # 1 = spam, 0 = ham

X_tr, X_te, y_tr, y_te = train_test_split(texts, labels, test_size=0.3, random_state=42)

pipe = Pipeline([
    ('vec', CountVectorizer()),
    ('nb', MultinomialNB(alpha=1.0)),
])
pipe.fit(X_tr, y_tr)
print(f"Test accuracy: {pipe.score(X_te, y_te):.4f}")

# New email predict
new = ["জরুরী আজকের লটারি জিতেছেন", "মিটিং আগামীকাল"]
preds = pipe.predict(new)
probs = pipe.predict_proba(new)
for t, p, pr in zip(new, preds, probs):
    label = "SPAM" if p == 1 else "HAM"
    print(f"\n{t!r}\n → {label}, P(spam)={pr[1]:.3f}")

Even tiny corpus — patterns capture। Spam keywords ("লটারি", "জিতেছেন", "জরুরী") — strong probability shift। Real production-এ লক্ষ-লক্ষ training emails।

৮ · কেন NB এত popular

Fast training: Single pass over data। Frequency count।
Fast inference: Few multiplications per prediction।
Memory efficient: Only counts/parameters needed।
Robust: Few hyperparameters (mainly $\alpha$)।
Calibration: Often surprisingly good for ranking।
Multi-class native: No OvR/OvO complexity।
Online learning friendly: Counts incrementally update।

৯ · কোথায় fail

Strong feature dependencies: "Naive" assumption violation severe হলে — biased probability estimates।
Continuous features non-Gaussian: Gaussian NB underperform।
Imbalanced data: Prior dominates — minority class miss।
Numerical features-এ scale-sensitive: Gaussian NB এর কম, Multinomial-এর বেশি।

Naive Bayes "starter" — সর্বদা baseline-এ run করুন। ৯০% production NLP — Naive Bayes-এর সাথে competitive performance প্রায়ই থাকে। Modern transformer যদি ৫% better — production cost benefit assess করুন।

ভাবনার প্রশ্ন

প্রতিটি প্রশ্ন নিজে কিছুক্ষণ ভাবুন — তারপর "→ উত্তর" চাপুন।

প্র ০১ "Naive" assumption clearly wrong — তবু কেন NB কাজ করে? Domingos & Pazzani-র "Optimality" paper কী দেখায়?

NB-র success — ML-এর একটি আকর্ষণীয় paradox।

Independence assumption violation:

Real features correlated।
Text — words co-occur।
Image — adjacent pixels correlated।
Bayes assumption strict false।

তবু NB কাজ করে কেন:

(১) Classification, not estimation:

Goal: $\arg\max P(y|\mathbf{x})$।
Probability values miscalibrated, কিন্তু ranking correct।
Class-এর order matters, exact value না।

(২) Domingos & Pazzani (1997):

"On the Optimality of the Simple Bayesian Classifier under Zero-One Loss"।
Independence violation → optimal classification possible।
If posteriors-এর order preserve হয় → correct prediction।

(৩) Decision boundary preservation:

Probability values ভুল হলেও — decision boundary same।
$P(\text{spam}) = 0.6$ vs true $0.8$ — both classify spam।

(৪) Bias variance tradeoff:

NB high bias (wrong assumption)।
Low variance (few parameters)।
Overall MSE often acceptable।

Empirical evidence:

Lewis (1998) — text classification benchmark।
NB competitive with SVM for many tasks।
Especially with small training data।
Robust to noisy labels।

When NB excels:

Text classification: Bag-of-words sparse — independence less violated।
Sentiment analysis: Strong word-class associations।
Medical diagnosis: Symptom-disease relatively independent।
Sparse data: Other models overfit।

When NB fails:

Strongly correlated features: Image classification।
Calibrated probabilities needed: Risk assessment।
Complex interactions: Higher-order features।

"Optimal" NB conditions:

Independence approximately holds।
Features informative individually।
Linear decision boundary adequate।

Theoretical extensions:

TAN (Tree-Augmented NB): Limited dependencies।
BAN (Bayesian Augmented NB): More structure।
HNB (Hidden NB): Latent variables।
Trade simplicity for accuracy।

Modern perspective:

NB still strong baseline।
"Don't be fancy first"।
Compare against NB before complex models।
Neural networks often only marginally better।

Calibration issue:

NB probabilities — typically extreme (close to 0 or 1)।
Independence violation → multiplied effects।
Solution: Platt scaling, isotonic regression।

Connection to regularization:

Strong prior assumptions — implicit regularization।
Few parameters → robust।
Bayesian framework — natural prior।

Modern hybrid:

NB as feature for downstream models।
NB ensemble component।
Boosted NB।

মূল উপলব্ধি: "Wrong but useful" — NB-র epitome। Decision-focused ML-এ exact probability-র চেয়ে decision quality matter। Engineering pragmatism over theoretical purity।

প্র ০২ "Laplace smoothing" কী? Zero-frequency problem-এর consequence কী? Bangla NLP-এ কেন crucial?

Smoothing — NB-র implementation detail কিন্তু production-এ critical।

Zero-frequency problem:

Test email: "জরুরী মিটিং"।
Training-এ "জরুরী" সব ham।
$P(\text{জরুরী} | \text{spam}) = 0$।
Product = 0 — হন্যা multiplicative effect।
মডেল decisive কিন্তু ভুল।

Laplace (add-one) smoothing:

$$P(w | c) = \frac{N_{w,c} + 1}{\sum_{w'} N_{w',c} + V}$$

Numerator — actual count + 1।
Denominator — total count + vocabulary size।
"Add one" to every count।

Generalized — add-$\alpha$:

$\alpha = 1$ — standard Laplace।
$\alpha = 0.1$ — Lidstone smoothing।
$\alpha = 0$ — no smoothing।
scikit-learn default $\alpha = 1$।

Bayesian interpretation:

Dirichlet prior on probabilities।
$\alpha$ — prior strength।
$\alpha$ small — data dominates।
$\alpha$ large — uniform prior dominates।

Bangla NLP specific:

(১) Inflectional richness:

Bangla words — many forms (verb conjugation, noun cases)।
"করা", "করি", "করেন", "করেছিল" — same root।
Each form rare in training।
Smoothing critical।

(২) Code-switching:

Bangla + English mixed।
Vocabulary explodes।
Many words seen only once।
Smoothing prevents collapse।

(৩) Romanization variation:

"valo", "bhalo", "ভালো" — same word।
Multiple representations।
Each underseen।

(৪) Domain-specific:

Medical, legal, technical — specialized vocabulary।
General corpus inadequate।
Smoothing handle unseen।

Alternative smoothing:

(১) Good-Turing:

Frequency of frequencies।
"Once-seen" probability redistributed।
Statistical theory rich।
Implementation complex।

(২) Kneser-Ney:

N-gram language models standard।
Discounting + back-off।
Beyond NB scope।

(৩) Witten-Bell:

Number of distinct items observed।
Document classification।

$\alpha$ tuning:

Cross-validation।
$\alpha \in \{0.01, 0.1, 1, 10\}$ try।
Smaller corpus → larger $\alpha$।
Domain-specific data → smaller $\alpha$।

Empirical findings:

Laplace ($\alpha = 1$) almost always works।
Smaller $\alpha$ — slight improvement large data।
$\alpha = 0$ disastrous on rare words।

Implementation:

scikit-learn — built-in।
Custom — log space critical।
Numerical stability।

Beyond smoothing:

Subword tokenization — BPE।
Character-level models।
Embeddings — semantic generalization।

Modern relevance:

NB still NLP baseline।
Bangla NLP — limited resources।
Laplace standard practice।

মূল উপলব্ধি: Laplace smoothing — simple yet vital। Zero-frequency catastrophic without। Bangla NLP — language structure makes smoothing especially important। Implementation detail, production criticality।

প্র ০৩ Spam detection — Naive Bayes vs। modern Transformer। Production-এ trade-offs কী? কখন NB-ই keep করবেন?

Classic vs modern — production decision matrix।

NB strengths in spam detection:

Lightning fast (microseconds)।
Tiny memory footprint।
Online learning natural।
Interpretable — keyword analysis।
Handles imbalanced classes okay।
Per-domain customization easy।

NB weaknesses:

Misses subtle context।
Can be tricked by stuffing words।
"Bag of words" — sequence ignored।
Calibration off।

Transformer strengths:

Context-aware।
Sequence understanding।
Adversarial-robust (more)।
Multi-language native।
Higher accuracy typically।

Transformer weaknesses:

Compute-intensive।
GPU often needed।
Higher latency (10-100ms)।
Bigger model — deployment complex।
Training data requirement larger।
Black-box interpretation।

Use case analysis:

(১) Email volume:

Gmail-scale: billions/day।
Per-email cost matters massively।
NB layer first, ensemble later।

(২) Real-time requirement:

Strict <10ms — NB winning।
~100ms okay — Transformer।
Async processing — flexibility।

(৩) Adaptation speed:

New spam pattern — retrain NB minutes।
Transformer fine-tune hours।
Spammer arms race favors NB।

(৪) Privacy:

NB on-device feasible।
Transformer often cloud।
Email content privacy critical।

Hybrid architecture:

(১) Cascaded:

NB first pass — most clear cases।
Transformer for borderline।
Cost-effective।

(২) Ensemble:

NB + LR + Transformer voting।
Weighted combination।
Robust।

(৩) Feature extraction:

NB outputs — feature for NN।
Best of both।

Production stack — modern email:

SPF/DKIM — sender verification।
Heuristic rules — clear bad senders।
NB — fast filter।
Gradient boosting — feature engineering।
Neural — borderline cases।
Human-in-loop — feedback।

Bangladesh context:

Local language spam (Bangla)।
Limited training data Transformer-এর জন্য।
NB shines with small domain data।
Mobile carrier SMS spam — NB common।

Cost analysis:

NB: $0.001/1k emails।
Transformer: $0.10/1k emails।
100x cost difference।
Volume × cost = total impact।

Accuracy comparison:

NB baseline: ৯৭-৯৮%।
Transformer: ৯৯%।
Marginal lift, massive cost।
Diminishing returns।

When stick with NB:

High volume + cost sensitivity।
Interpretability requirement।
Limited training data।
Frequent retraining needed।
On-device deployment।

When upgrade to Transformer:

Adversarial spam evolving।
Subtle pattern detection critical।
Multi-lingual content।
Compute budget allows।

মূল উপলব্ধি: NB ও Transformer — competing tools, complementary strengths। Production decision rarely "best model"; usually "right tool for cost/performance balance"। NB still alive in 2026।

প্র ০৪ "Generative vs discriminative" — NB generative, logistic regression discriminative। কোন setting-এ কোনটি favor করে?

ML-এর foundational dichotomy — model design choice।

Definitions:

Generative: $P(\mathbf{x}, y)$ model — joint distribution।
Discriminative: $P(y | \mathbf{x})$ model — conditional।

NB — generative:

$P(\mathbf{x} | y)$ explicitly model।
$P(y)$ explicitly model।
Bayes rule → $P(y | \mathbf{x})$।
Can generate samples (theoretically)।

Logistic regression — discriminative:

Direct $P(y | \mathbf{x})$ model।
$P(\mathbf{x})$ ignored।
Cannot generate samples।
Decision boundary direct।

Ng & Jordan (2001) classic paper:

"On Discriminative vs Generative Classifiers"।
Theoretical comparison NB vs LR।
Asymptotically — LR (or equivalent discriminative) better।
Small data — NB often wins।

Sample complexity:

NB: $O(\log n)$ samples to converge।
LR: $O(n)$ samples।
$n$ — feature dimensionality।
NB faster convergence।

Asymptotic accuracy:

LR — better in limit।
NB — independence assumption ceiling।
Crossover point — depends।

Generative advantages:

(১) Small data:

Strong assumptions help।
Less overfitting।
Quick convergence।

(২) Missing data:

$P(\mathbf{x}, y)$ — marginalize unknown variables।
Discriminative — harder।

(৩) Anomaly detection:

$P(\mathbf{x})$ low — anomaly।
One-class classification।
Discriminative awkward।

(৪) Structured prediction:

HMM, CRF — sequence models।
Generative natural fit।

(৫) Sample generation:

Data augmentation।
Synthetic data।
Privacy preservation।

Discriminative advantages:

(১) Big data:

Direct optimization of relevant quantity।
Better asymptotic accuracy।

(২) Conditional dependencies:

Feature correlations modeled directly।
No "naive" assumption।

(৩) Computational efficiency:

Don't waste capacity on $P(\mathbf{x})$।
Focus on decision boundary।

Modern landscape:

(১) Deep generative models:

VAE, GAN, Diffusion।
Generative renaissance।
$P(\mathbf{x})$ critical (image generation)।

(২) Discriminative dominance:

Classification — Transformer, BERT।
$P(y|\mathbf{x})$ direct fit।

(৩) Hybrid:

Pre-train generative (LM)।
Fine-tune discriminative (classification)।
Best of both।

NB-LR equivalence under certain conditions:

Gaussian NB with shared covariance — LDA।
LDA — linear discriminant analysis।
Decision boundary identical to LR।
Under specific assumptions।

Choice in practice:

(১) Decision criteria:

Data size: small → NB, large → LR/NN।
Compute: limited → NB।
Interpretation: NB or LR — both interpretable।
Anomaly detection: generative।
Generation needed: generative obviously।

(২) Default choice:

Quick baseline: NB।
Production classifier: LR (after baseline)।
State-of-art: NN-based discriminative।