মডেলের সফলতা কীভাবে মাপি

এই subtlety — ML-এ underrated অথচ critical. ROC-AUC সব দেখলে production AI অনেক ভুলে পড়ে।

(১) ROC Curve বনাম PR Curve:

• ROC: TPR (Recall) vs FPR — y-axis recall, x-axis false alarm rate.
• PR: Precision vs Recall — y-axis precision, x-axis recall.
• দু'টি curve = same data, different perspective.

(২) Imbalance-এর প্রভাব:

• ৯৯% Negative, ১% Positive (rare disease)।
• FPR = FP/(FP+TN) — denominator-এ TN বিশাল।
• FP যতই বাড়ুক — FPR ছোট থাকে (TN dominant)।
• ROC curve "ভাল" দেখায় — false sense of security.

(৩) Concrete উদাহরণ:

• Dataset: ১,০০০ negative, ১০ positive.
• Model predicts top 50 as positive: ৫টি correct, ৪৫টি wrong.
• FPR = 45/1000 = 0.045 — looks good.
• Precision = 5/50 = 0.10 — looks bad!
• একই model, দু'টি দৃষ্টি।

(৪) যখন ROC-AUC ভাল:

• Balanced classes (50-50)।
• Both classes equally important.
• Standard benchmark comparison.

(৫) যখন PR-AUC জরুরি:

• Imbalanced (rare positive class)।
• Positive class বেশি important.
• Information retrieval (search, recommendations)।
• Anomaly/fraud detection.
• Disease detection (rare condition)।

(৬) Random baseline:

• ROC: random model = 0.5 (universal)।
• PR: random = positive class proportion (varies!)।
• ৫% positive → random PR-AUC = 0.05 only.
• PR-AUC 0.30 = ৬x improvement (impressive)।

(৭) Visualization clarity:

• ROC curve quickly approaches top-left → overoptimistic.
• PR curve dramatic shape changes → real trade-off visible.
• Stakeholder communication-এ PR clearer.

(৮) Production decision making:

• Threshold selection on PR curve.
• "How many false alarms can we tolerate?" → precision target.
• "How many cases can we miss?" → recall target.
• F-beta score: $F_\beta = (1+\beta^2) \cdot \frac{P \cdot R}{\beta^2 P + R}$।
• $\beta = 2$: recall 2x weight; $\beta = 0.5$: precision 2x.

(৯) Practical scenarios:

• Spam filter: PR-AUC critical (don't lose important mails)।
• Medical screening: Both — maximize recall under precision constraint.
• Recommendation: PR-AUC at top-k (precision@10)।
• Search: Precision@k, NDCG.

(১০) Calibration matters:

• Probability score-গুলো actually probability নাই হতে পারে।
• Reliability diagram check.
• Calibrated হলে — threshold selection meaningful.

(১১) Common mistakes:

• Reporting only ROC-AUC on imbalanced data.
• Comparing models across different class balances.
• Ignoring per-class metrics.
• Single point on curve (single threshold) only.

(১২) Bangladesh use cases:

• Health screening (TB, cancer): PR-AUC primary, ROC secondary.
• Loan default: Imbalanced, PR-AUC + business cost-aware.
• Pirated content detection: Few positives, PR critical.
• Crop disease: Rare disease classes, macro-F1.

মূল উপলব্ধি: Single number metric — illusion. Imbalanced ডেটায় ROC-AUC misleading, PR-AUC honest. Always context-aware metric choice. Senior ML engineer = knows which to trust when.

এই question — ML engineer-এর maturity test. জুনিয়র "wow" বলে; senior interrogate করেন।

প্রশ্ন ১: "Class distribution কী?"

• ৯২% accuracy + ৯২% majority class = useless model.
• Class imbalance prove or refute.
• Always confusion matrix demand.

প্রশ্ন ২: "Test set কেমন? কতবার দেখেছ?"

• Hold-out test? Cross-validation?
• Single eval বা multiple?
• Test set leakage check হয়েছে?
• Group/time split correct?

প্রশ্ন ৩: "Per-class performance কী?"

• Class A: 99%, Class B: 50% — overall 92% misleading.
• Per-class precision, recall, F1.
• Failure modes identify.

প্রশ্ন ৪: "Baseline-এর সাথে compare করেছ?"

• Random predictor: ৫০%।
• Always-majority: ৭০%? ৮০%?
• Simple logistic regression baseline?
• Improvement Δ — meaningful?

প্রশ্ন ৫: "Production environment কেমন?"

• Latency requirement?
• Memory/cost?
• Real-world distribution = test distribution?
• Domain shift expected?

আরও গভীর প্রশ্ন:

• "Confidence interval কী?" — point estimate vs range.
• "Calibration কেমন?" — probability scores trustworthy?
• "Bias subgroups-এ কেমন?" — fairness check.
• "Adversarial robustness?" — small perturbation-এ কেমন?
• "Failure mode catalog?" — কোথায় কেন ভুল?

Common red flags:

• "৯৯% accuracy" — leakage সন্দেহ।
• "একটি single number" — incomplete picture.
• "Baseline নাই" — improvement unmeasured.
• "Test set ৫০ samples" — variance high.
• "Train accuracy ৯৯%, test ৯২%" — overfit warning.

Stakeholder communication:

• Boss-এর কাছে যাবার আগে — পুরো picture প্রস্তুত।
• Confusion matrix, per-class metrics, baseline comparison.
• Honest discussion of limitations.
• Future improvement plan.

Practical communication:

• "৯২% overall accuracy" — start with context.
• "Class A: 95%, B: 88%" — granular.
• "Baseline 70%, our 92% — 22% absolute improvement"।
• "Test on 10,000 examples, 95% CI: 91.5-92.5%"।
• Deployment recommendation.

Anti-pattern:

• "৯২% — let's deploy!" — premature.
• Single experiment, no replication.
• Cherry-picked test set.
• Hyperparameter tuned on test set.

মূল উপলব্ধি: Single number — ML communication-এর সবচেয়ে dangerous habit. ৯২% — meaningless without context. Senior engineer পাঁচটি query চালিয়ে decompose করেন। Junior to senior transition = thinking in distributions, not points.

Calibration — production AI safety-র key topic. Confidence score actually probability হলেই decision-making সুন্দর। Modern DL এই দিকে problematic.

(১) Calibration কী?

• Model predicts 80% confidence → 80% of those should actually be positive.
• Perfect calibration: prediction = actual probability.
• Miscalibration: 90% confidence কিন্তু actual 60% — overconfident.

(২) কেন important?

• Decision making — confidence threshold-এ depend করে।
• Risk assessment — wrong probability = wrong risk.
• Human-AI collaboration — trust calibrated AI.
• Uncertainty quantification — production critical.

(৩) Reliability Diagram:

• x-axis: predicted probability bin.
• y-axis: actual fraction positive.
• Diagonal = perfect calibration.
• Above = underconfident, below = overconfident.

(৪) Expected Calibration Error (ECE):

• Bins-এ predictions ভাগ।
• Each bin: |avg predicted - avg actual|।
• Weighted by bin size.
• 0 = perfect, higher = worse.

(৫) Modern DL overconfident কেন?

• Cross-entropy loss পুশ করে probability extremes-এ (0 or 1)।
• Network capacity বিশাল — train data-এ overfit.
• Batch normalization, dropout — calibration disturb.
• Guo et al. (2017): "On Calibration of Modern Neural Networks" — ResNet, DenseNet সবই miscalibrated.

(৬) Calibration techniques:

• Platt Scaling:
  • Sigmoid fit on logits.
  • $P = \sigma(a \cdot z + b)$।
  • Validation set-এ optimize.
• Temperature Scaling:
  • Single parameter $T$।
  • $P_i = \text{softmax}(z_i / T)$।
  • $T > 1$: smoother distribution.
  • Simple, effective for DL.
• Isotonic Regression:
  • Non-parametric monotonic mapping.
  • More flexible than Platt.
  • Need more data.

(৭) Implementation:

import numpy as np
from scipy.optimize import minimize

# Temperature scaling
def cross_entropy(T, logits, labels):
    probs = softmax(logits / T)
    return -np.mean(np.log(probs[range(len(labels)), labels] + 1e-9))

result = minimize(cross_entropy, x0=1.0, args=(val_logits, val_labels))
T = result.x[0]
# Apply T at inference time

(৮) Beyond Calibration — Uncertainty:

• Aleatoric: data noise — irreducible.
• Epistemic: model uncertainty — reducible with data.
• MC Dropout — কম খরচে Bayesian approximation.
• Ensemble — multiple models, variance = uncertainty.

(৯) Production examples:

• Self-driving: brake decision threshold calibrated.
• Medical: triage based on confidence.
• Loan approval: probability of default = expected loss.
• Content moderation: high-confidence auto-action, low → human review.

(১০) Conformal Prediction:

• Distribution-free uncertainty.
• "95% prediction interval" guaranteed.
• Modern alternative to calibration.

(১১) LLM context:

• Hallucination = miscalibrated confidence.
• "Self-confidence" assessment hard.
• Anthropic, OpenAI research: instruct LLM to express uncertainty.
• Token probability ≠ semantic certainty.

(১২) Bangladesh context:

• Medical AI: calibration কঠিনভাবে required.
• Bangla NLP: limited data — uncertainty wider.
• Public-facing AI: "I don't know" allowed কি না?
• Educational tool: confidence display students-কে।

(১৩) Common mistakes:

• Threshold selection without calibration check.
• Probability-as-truth in downstream pipelines.
• Calibration on test set (cheating)।
• Single calibration globally — should be per-class.

মূল উপলব্ধি: Confident wrong > uncertain right — false. Calibration = honest AI. ৯০% confidence-এ ৯০% accuracy — production trust-এর foundation. Modern DL miscalibration default — fix-এর effort price-of-deployment.

Algorithmic fairness — modern AI ethics-এর core. Production AI deploy করতে গেলে fairness audit required (অনেক jurisdiction-এ legal mandate)।

(১) Bias-এর source:

• Historical bias: Past data-এ society-এর bias embedded.
• Sampling bias: কিছু group underrepresented.
• Label bias: Annotators-এর implicit bias.
• Algorithmic bias: Optimization artifacts.
• Deployment bias: Different group-এ different usage pattern.

(২) Fairness Metrics — Group:

• Demographic Parity:
  • $P(\hat{Y}=1 | A=0) = P(\hat{Y}=1 | A=1)$।
  • সব group-এ equal positive rate.
  • সমস্যা: ground truth ভিন্ন হলে problematic.
• Equal Opportunity:
  • $P(\hat{Y}=1 | Y=1, A=0) = P(\hat{Y}=1 | Y=1, A=1)$।
  • সব group-এ equal recall.
  • True positive-এ fairness.
• Equalized Odds:
  • Equal opportunity + equal FP rate.
  • Stronger constraint.
• Calibration parity:
  • সব group-এ same calibration.
  • Probability score equally meaningful.

(৩) Individual Fairness:

• "Similar individuals → similar predictions"।
• Distance-based formalization.
• Group fairness-এর সাথে tension.

(৪) Impossibility theorem:

• Chouldechova (২০১৭), Kleinberg (২০১৬): সব fairness criteria simultaneously satisfy অসম্ভব।
• Trade-off explicit হতে হবে।
• Context-specific choice.

(৫) Detection workflow:

1. Protected attributes identify (age, gender, location, religion)।
2. Per-group metrics calculate.
3. Statistical significance test.
4. Practical significance assess.
5. Disaggregated reporting.

(৬) Tools:

• Fairlearn (Microsoft): Python library.
• AI Fairness 360 (IBM): Comprehensive toolkit.
• What-If Tool (Google): Interactive analysis.
• Aequitas: Bias audit reports.

(৭) Mitigation strategies:

• Pre-processing:
  • Reweighting samples.
  • Disparate impact remover.
  • Sampling strategies.
• In-processing:
  • Fairness-constrained optimization.
  • Adversarial debiasing.
  • Multi-objective loss.
• Post-processing:
  • Threshold adjustment per group.
  • Calibration per group.
  • Output transformation.

(৮) Famous failure cases:

• COMPAS (criminal recidivism): black defendants-এর FP rate বেশি।
• Amazon hiring: men preferred (২০১৮ scrapped)।
• Apple Card: women lower credit limits.
• Healthcare AI: less care recommended for black patients.

(৯) LLM fairness:

• Stereotype amplification.
• Toxicity towards minorities.
• Representation gaps.
• RLHF helps but doesn't solve.

(১০) Bangladesh-specific concerns:

• Religion bias:
  • Name-based discrimination.
  • Loan approval, hiring.
  • Sensitive attribute → বিশেষ care.
• Gender:
  • Workplace AI — female underrepresentation.
  • Health AI — pregnancy bias.
• Geographic:
  • Dhaka vs rural — service quality.
  • Hill tracts populations underrepresented.
• Disability:
  • Accessibility consideration.
  • Speech recognition — speech impairment.
• Socioeconomic:
  • Education level proxies.
  • Access to digital services.

(১১) Legal landscape:

• EU AI Act (২০২৪): high-risk AI fairness audit required.
• US: Equal Credit Opportunity Act, Fair Housing Act.
• Bangladesh: Data Protection Act ২০২৩।
• Industry standards evolving.

(১২) Practical advice:

• Diverse team — different perspectives catch bias.
• Stakeholder consultation.
• Continuous monitoring (not one-time check)।
• Transparent reporting.
• Easy redress for affected users.

(১৩) Future directions:

• Causal fairness frameworks.
• Counterfactual fairness.
• Long-term fairness dynamics.
• Participatory ML — community involvement.

মূল উপলব্ধি: Accuracy alone insufficient. Disaggregated, fair-aware metrics — production AI minimum. বাংলাদেশের সমাজ-বৈচিত্র্যে — bias detection extra critical. ভাল ML engineer accuracy চান, দারুণ ML engineer accuracy + fairness চান। নৈতিকতা = engineering quality.