পাঠ ২৪ · ৩৩-এর মধ্যে · মডিউল ৪
Home / AI Courses / MLOps / Model monitoring

Model monitoring কী, কেন

Model monitoring — what & why
৭ মিনিট পড়া মধ্য · Intermediate Concept

এই পাঠে যা শিখবেন

  • ৪-layer monitoring architecture
  • Ground truth lag handling
  • Alert design — precision, escalation
  • Observability vs monitoring

১ · "Model alive কিন্তু dying"

Production ML-এর সবচেয়ে বিপজ্জনক failure — silent. API 200 OK return করছে, latency <100ms, error rate 0%. কিন্তু prediction quality ধীরে ধীরে কমছে।

  • 2024 মে: model accuracy 92%.
  • 2024 ডিসেম্বর: 76% (drift unmonitored).
  • 2025 জানুয়ারি: business team দেখল conversion রেট অর্ধেকে।

২ · ৪-layer monitoring

৪ স্তর

১) Infrastructure: CPU, RAM, latency, error rate.
২) Data quality: input schema, missing, range, distribution.
৩) Model performance: accuracy (যদি ground truth available), prediction distribution.
৪) Business KPI: conversion, fraud rate, retention.

৩ · Layer 1: Infrastructure

  • HTTP latency p50/p95/p99.
  • Request rate (RPS).
  • Error rate (4xx, 5xx).
  • CPU, RAM, GPU utilization.
  • Pod restart count.

Tools: Prometheus + Grafana (Lesson 27). Standard DevOps. ML-অ-aware.

৪ · Layer 2: Data quality

  • Schema: expected columns present?
  • Missing values: rate per column.
  • Range: age in [0,120], probability in [0,1].
  • Type: string vs numeric.
  • Cardinality: categorical unique count.
  • Distribution: mean, std, percentiles.

Tools: Great Expectations, Pandera, Evidently AI.

৫ · Layer 3: Model performance

  • Accuracy/AUC: ground truth-based; lag inevitable.
  • Prediction distribution: proxy — % positive predictions, mean prediction.
  • Confidence distribution: high vs low confidence rate.
  • Drift metric: PSI, KS (Lesson 25-26).

৬ · Layer 4: Business KPI

  • Conversion rate, click-through, retention.
  • Fraud chargeback rate.
  • Customer complaint per 1000.
  • Revenue per user.

Business KPI delayed — daily/weekly aggregate. But ultimate truth.

৭ · Ground truth lag

"Did model predict correctly?" — answer needs label. Many use cases label takes hours/days.

  • Click prediction — label in seconds (clicked or not).
  • Conversion — minutes-hours.
  • Fraud — days-months (chargeback dispute period).
  • Loan default — months-years.

Strategy:

  • Real-time proxies (drift, prediction distribution).
  • Labels arrive — async accuracy update.
  • Manual labeling sample for rapid feedback.

৮ · Observability vs monitoring

  • Monitoring: known unknowns — pre-defined dashboards, alerts.
  • Observability: unknown unknowns — explore, debug new issue.
  • ML observability: prediction logs queryable, lineage trace, root cause analysis.
  • Tools: Datadog, Honeycomb (general); Arize, WhyLabs, Fiddler (ML-specific).

৯ · Alerting design

  • Precision over recall: false alarm fatigue erode trust.
  • Tiered: page (critical), Slack (warn), digest (info).
  • Escalation: on-call → backup → manager.
  • Runbook: alert → action steps documented.
  • Periodic review: false positive rate, missed incident.
ML monitoring — 4 layers, increasing latency Infrastructure → Business KPI L1 Infrastructure: latency, RPS, error rate, CPU seconds L2 Data quality: schema, range, missing, distribution minutes L3 Model performance: drift (proxy), accuracy (lagged) hours L4 Business KPI: conversion, fraud rate, retention, complaints days Detection time: L1 fast (alert), L4 slow (real impact). সব layer needed.
৪ stratified monitoring layer — infrastructure থেকে business; signal-to-real-impact tradeoff.
বাংলাদেশ banking-এ ৪-layer monitoring Bangladesh Bank ICT guideline-এ বাধ্যতামূলক। Fintech early invest essential.

ভাবনার প্রশ্ন

প্র ০১"Ground truth never arrives — কীভাবে monitor?"

"Ground truth never arrives" common scenario — content moderation, recommendation, fraud edge cases.

Strategies:

  • Drift-based: input distribution shift → likely accuracy degradation.
  • Prediction distribution: "% positive" historical baseline সাথে compare.
  • Manual sample: daily 100 prediction human-label. Slow but real signal.
  • Indirect business signal: customer complaint per 1000 prediction.
  • A/B vs hold-out: small holdout never serve new model — long-term differential.

BD Daraz example:

  • Recommendation: clicked = label. Click within 5 min — label fast.
  • Item rejected by user: ambiguous — could be bad rec or not interest.
  • Solution: implicit signal hierarchy (click > view > skip).

মূল উপলব্ধি: Ground truth lag ≠ no signal. Proxies + manual sampling + drift monitoring layered. Imperfect signals combined → reliable picture.

প্র ০২"Alert fatigue — কীভাবে control?"

Alert fatigue erode trust.

Symptoms:

  • On-call ignore Slack notification.
  • "Resolve" without root cause analysis.
  • Real incident missed amid noise.

Strategies:

  • Threshold tuning: initial conservative; loosen over time per false-positive rate.
  • Time-based: "5-min sustained" not "instantaneous".
  • Composite: "latency AND error" both — single signal too noisy.
  • Suppress duplicate: 1 hour cooldown if same alert ringing.
  • Severity stratification: Page (P0), Slack alert (P1), email digest (P2).
  • Quarterly review: false positive % > 30% → tighten.

Anti-patterns:

  • "Alert on everything" — noise.
  • "Auto-acknowledge" — silent.
  • "This always alerts, ignore" — dangerous.

মূল উপলব্ধি: Alert quality > quantity. Tune ruthless, kill stale alerts. Trust earned by accuracy.

প্র ০৩"Observability tools — Datadog vs ML-specific (Arize)?"

General observability tool কি ML-specific tool replace করে?

Datadog/general:

  • Infrastructure + app monitoring great.
  • Custom metric ML add possible.
  • Single pane of glass.
  • BD context: many fintech use.

ML-specific (Arize, WhyLabs, Fiddler):

  • Drift built-in.
  • Feature-level monitoring.
  • Embedding/distribution analysis.
  • Bias detection.
  • Pricing: usually expensive enterprise.

Hybrid common:

  • Infrastructure → Datadog.
  • Drift, ML-specific → Evidently OSS or Arize.

BD reality:

  • Cost-conscious — Prometheus + Grafana + Evidently OSS dominant stack.
  • Enterprise SaaS expensive.

মূল উপলব্ধি: ML-specific tool nice-to-have, not essential. Custom Prometheus metrics + Evidently free coverage 80%. Specialized tool justified at scale.

প্র ০৪"Org-level monitoring maturity — assess + improve কীভাবে?"

Monitoring maturity stage-vise.

Stage 0 — None:

  • "Service alive কিনা ping check".
  • Issue user complaint-এ আসে।

Stage 1 — Infrastructure only:

  • Latency, error rate, CPU.
  • Standard DevOps.
  • ML-specific blind spot.

Stage 2 — + Data quality:

  • Schema validation, distribution check.
  • Pipeline-stage gates.

Stage 3 — + Drift:

  • PSI/KS continuous.
  • Concept drift detection.

Stage 4 — + Business KPI integration:

  • Model decisions → business outcome trace.
  • Long-term effect tracking.

Improvement plan:

  • Self-assess: "if production model accuracy degrades 10%, কত দিনে আমরা জানব?" — answer reveals stage.
  • Pick lowest-effort gap → quick win.
  • Quarterly progression.

BD typical:

  • Stage 1 most common — DevOps-heritage.
  • Bigger fintech Stage 2-3.
  • Stage 4 rare — even global standards.

মূল উপলব্ধি: Monitoring maturity gradual, intentional. Single big effort instead — prioritized layer-by-layer. Yearly stage upgrade realistic goal.

অনুশীলন

  1. Self-audit: আপনার (real or imagined) ML system — কোন stage-এ? Lowest-effort improvement?

    Stage 1 most common. Next: Great Expectations data validation (1 day), prediction distribution log (4 hours), drift PSI script (1 day).

  2. Alert design: ৩-tier alert plan — P0/P1/P2 examples.
    • P0 (page): error rate > 5%, p99 > 2× SLA.
    • P1 (Slack): drift > threshold, accuracy drop > 2%.
    • P2 (digest): minor schema warning, slow request count.
  3. চিন্তা: bKash fraud — ground truth chargeback 30-90 days. Real-time monitoring strategy?
    • Layer 1: latency, error.
    • Layer 2: input feature distribution drift (PSI on top features).
    • Layer 3: prediction distribution (% high-risk flagged).
    • Layer 4: chargeback rate (lagged but conclusive).
    • Manual sample 100/day expert review for rapid feedback.

আরও পড়ুন

এই পাঠে ভুল পেয়েছেন? জানান
পূর্ববর্তী পাঠ
পাঠ ২৩ · A/B testing