Model monitoring কী, কেন
এই পাঠে যা শিখবেন
- ৪-layer monitoring architecture
- Ground truth lag handling
- Alert design — precision, escalation
- Observability vs monitoring
১ · "Model alive কিন্তু dying"
Production ML-এর সবচেয়ে বিপজ্জনক failure — silent। API 200 OK return করছে, latency <100ms, error rate 0%। কিন্তু prediction quality ধীরে ধীরে কমছে।
- 2024 মে: model accuracy 92%।
- 2024 ডিসেম্বর: 76% (drift unmonitored)।
- 2025 জানুয়ারি: business team দেখল conversion রেট অর্ধেকে।
২ · ৪-layer monitoring
১) Infrastructure: CPU, RAM, latency, error rate।
২) Data quality: input schema, missing, range, distribution।
৩) Model performance: accuracy (যদি ground truth available), prediction distribution।
৪) Business KPI: conversion, fraud rate, retention।
৩ · Layer 1: Infrastructure
- HTTP latency p50/p95/p99।
- Request rate (RPS)।
- Error rate (4xx, 5xx)।
- CPU, RAM, GPU utilization।
- Pod restart count।
Tools: Prometheus + Grafana (Lesson 27)। Standard DevOps। ML-অ-aware।
৪ · Layer 2: Data quality
- Schema: expected columns present?
- Missing values: rate per column।
- Range: age in [0,120], probability in [0,1]।
- Type: string vs numeric।
- Cardinality: categorical unique count।
- Distribution: mean, std, percentiles।
Tools: Great Expectations, Pandera, Evidently AI।
৫ · Layer 3: Model performance
- Accuracy/AUC: ground truth-based; lag inevitable।
- Prediction distribution: proxy — % positive predictions, mean prediction।
- Confidence distribution: high vs low confidence rate।
- Drift metric: PSI, KS (Lesson 25-26)।
৬ · Layer 4: Business KPI
- Conversion rate, click-through, retention।
- Fraud chargeback rate।
- Customer complaint per 1000।
- Revenue per user।
Business KPI delayed — daily/weekly aggregate। But ultimate truth।
৭ · Ground truth lag
"Did model predict correctly?" — answer needs label। Many use cases label takes hours/days।
- Click prediction — label in seconds (clicked or not)।
- Conversion — minutes-hours।
- Fraud — days-months (chargeback dispute period)।
- Loan default — months-years।
Strategy:
- Real-time proxies (drift, prediction distribution)।
- Labels arrive — async accuracy update।
- Manual labeling sample for rapid feedback।
৮ · Observability vs monitoring
- Monitoring: known unknowns — pre-defined dashboards, alerts।
- Observability: unknown unknowns — explore, debug new issue।
- ML observability: prediction logs queryable, lineage trace, root cause analysis।
- Tools: Datadog, Honeycomb (general); Arize, WhyLabs, Fiddler (ML-specific)।
৯ · Alerting design
- Precision over recall: false alarm fatigue erode trust।
- Tiered: page (critical), Slack (warn), digest (info)।
- Escalation: on-call → backup → manager।
- Runbook: alert → action steps documented।
- Periodic review: false positive rate, missed incident।
ভাবনার প্রশ্ন
প্র ০১"Ground truth never arrives — কীভাবে monitor?"
"Ground truth never arrives" common scenario — content moderation, recommendation, fraud edge cases।
Strategies:
- Drift-based: input distribution shift → likely accuracy degradation।
- Prediction distribution: "% positive" historical baseline সাথে compare।
- Manual sample: daily 100 prediction human-label। Slow but real signal।
- Indirect business signal: customer complaint per 1000 prediction।
- A/B vs hold-out: small holdout never serve new model — long-term differential।
BD Daraz example:
- Recommendation: clicked = label। Click within 5 min — label fast।
- Item rejected by user: ambiguous — could be bad rec or not interest।
- Solution: implicit signal hierarchy (click > view > skip)।
মূল উপলব্ধি: Ground truth lag ≠ no signal। Proxies + manual sampling + drift monitoring layered। Imperfect signals combined → reliable picture।
প্র ০২"Alert fatigue — কীভাবে control?"
Alert fatigue erode trust।
Symptoms:
- On-call ignore Slack notification।
- "Resolve" without root cause analysis।
- Real incident missed amid noise।
Strategies:
- Threshold tuning: initial conservative; loosen over time per false-positive rate।
- Time-based: "5-min sustained" not "instantaneous"।
- Composite: "latency AND error" both — single signal too noisy।
- Suppress duplicate: 1 hour cooldown if same alert ringing।
- Severity stratification: Page (P0), Slack alert (P1), email digest (P2)।
- Quarterly review: false positive % > 30% → tighten।
Anti-patterns:
- "Alert on everything" — noise।
- "Auto-acknowledge" — silent।
- "This always alerts, ignore" — dangerous।
মূল উপলব্ধি: Alert quality > quantity। Tune ruthless, kill stale alerts। Trust earned by accuracy।
প্র ০৩"Observability tools — Datadog vs ML-specific (Arize)?"
General observability tool কি ML-specific tool replace করে?
Datadog/general:
- Infrastructure + app monitoring great।
- Custom metric ML add possible।
- Single pane of glass।
- BD context: many fintech use।
ML-specific (Arize, WhyLabs, Fiddler):
- Drift built-in.
- Feature-level monitoring।
- Embedding/distribution analysis।
- Bias detection।
- Pricing: usually expensive enterprise।
Hybrid common:
- Infrastructure → Datadog।
- Drift, ML-specific → Evidently OSS or Arize।
BD reality:
- Cost-conscious — Prometheus + Grafana + Evidently OSS dominant stack।
- Enterprise SaaS expensive।
মূল উপলব্ধি: ML-specific tool nice-to-have, not essential। Custom Prometheus metrics + Evidently free coverage 80%। Specialized tool justified at scale।
প্র ০৪"Org-level monitoring maturity — assess + improve কীভাবে?"
Monitoring maturity stage-vise।
Stage 0 — None:
- "Service alive কিনা ping check"।
- Issue user complaint-এ আসে।
Stage 1 — Infrastructure only:
- Latency, error rate, CPU।
- Standard DevOps।
- ML-specific blind spot।
Stage 2 — + Data quality:
- Schema validation, distribution check।
- Pipeline-stage gates।
Stage 3 — + Drift:
- PSI/KS continuous।
- Concept drift detection।
Stage 4 — + Business KPI integration:
- Model decisions → business outcome trace।
- Long-term effect tracking।
Improvement plan:
- Self-assess: "if production model accuracy degrades 10%, কত দিনে আমরা জানব?" — answer reveals stage।
- Pick lowest-effort gap → quick win।
- Quarterly progression।
BD typical:
- Stage 1 most common — DevOps-heritage।
- Bigger fintech Stage 2-3।
- Stage 4 rare — even global standards।
মূল উপলব্ধি: Monitoring maturity gradual, intentional। Single big effort instead — prioritized layer-by-layer। Yearly stage upgrade realistic goal।
অনুশীলন
- Self-audit: আপনার (real or imagined) ML system — কোন stage-এ? Lowest-effort improvement?
Stage 1 most common। Next: Great Expectations data validation (1 day), prediction distribution log (4 hours), drift PSI script (1 day)।
- Alert design: ৩-tier alert plan — P0/P1/P2 examples।
- P0 (page): error rate > 5%, p99 > 2× SLA।
- P1 (Slack): drift > threshold, accuracy drop > 2%।
- P2 (digest): minor schema warning, slow request count।
- চিন্তা: bKash fraud — ground truth chargeback 30-90 days। Real-time monitoring strategy?
- Layer 1: latency, error।
- Layer 2: input feature distribution drift (PSI on top features)।
- Layer 3: prediction distribution (% high-risk flagged)।
- Layer 4: chargeback rate (lagged but conclusive)।
- Manual sample 100/day expert review for rapid feedback।