পাঠ ২৭ · ৩৩-এর মধ্যে · মডিউল ৪

Home / AI Courses / MLOps / Prometheus & Grafana

Prometheus ও Grafana

Prometheus + Grafana for ML monitoring

৭ মিনিট পড়া মধ্য · Intermediate Tooling

এই পাঠে যা শিখবেন

Prometheus pull model + service discovery
৪ metric types — কোনটা কখন
FastAPI-এ ML metrics instrumentation
PromQL basics + Grafana dashboard

১ · Pull-based architecture

Prometheus standard pull model — service expose HTTP /metrics; Prometheus periodically scrape (15-60 sec interval)।

Pros: no metric server-overload (scrape rate Prometheus-controlled)।
Pros: service-discovery friendly (K8s, Consul)।
Cons: short-lived job tricky (Pushgateway alternative)।

২ · Metric types

Counter: monotonically increasing — request count, error count।
Gauge: goes up/down — queue depth, model accuracy।
Histogram: distribution — latency p50/p95/p99 observable।
Summary: pre-computed quantile — like histogram but server-side।

৩ · FastAPI-এ ML metrics

Python · prometheus-fastapi-instrumentator

from fastapi import FastAPI
from prometheus_fastapi_instrumentator import Instrumentator
from prometheus_client import Counter, Histogram, Gauge

app = FastAPI()

# ML-specific custom metrics
predictions_total = Counter(
    "ml_predictions_total",
    "Total predictions made",
    ["model_version", "label"],
)
prediction_confidence = Histogram(
    "ml_prediction_confidence",
    "Confidence distribution",
    ["model_version"],
    buckets=[0.1, 0.3, 0.5, 0.7, 0.9, 1.0],
)
inference_latency = Histogram(
    "ml_inference_latency_seconds",
    "Model inference time",
    ["model_version"],
    buckets=[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0],
)
data_drift_psi = Gauge(
    "ml_data_drift_psi",
    "Population Stability Index per feature",
    ["feature"],
)

# Auto-instrument HTTP metrics
Instrumentator().instrument(app).expose(app)

@app.post("/predict")
def predict(req):
    import time
    start = time.perf_counter()
    proba = model.predict_proba([req.text])[0]
    label = str(proba.argmax())
    confidence = float(proba.max())

    elapsed = time.perf_counter() - start
    predictions_total.labels(
        model_version=MODEL_VERSION, label=label
    ).inc()
    prediction_confidence.labels(
        model_version=MODEL_VERSION
    ).observe(confidence)
    inference_latency.labels(
        model_version=MODEL_VERSION
    ).observe(elapsed)

    return {"label": label, "confidence": confidence}

# /metrics endpoint auto-exposed by Instrumentator

ML-specific metrics: prediction count by label (distribution skew), confidence histogram (drift signal), per-version latency। Drift PSI separate scrape job from data validation pipeline।

৪ · Prometheus config

YAML · prometheus.yml

global:
  scrape_interval: 30s

scrape_configs:
  - job_name: 'ml-services'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: '.*ml-.*'
        action: keep

  - job_name: 'sentiment-api'
    static_configs:
      - targets: ['sentiment-api:8000']

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

rule_files:
  - 'ml_alerts.yml'

K8s service discovery — pod label-based filter। Alertmanager-এ rules forward।

৫ · PromQL basics

rate(http_requests_total[5m]) — RPS over 5 min।
histogram_quantile(0.99, rate(ml_inference_latency_seconds_bucket[5m])) — p99 latency।
sum by (model_version) (rate(ml_predictions_total[5m])) — RPS per version।
avg_over_time(ml_data_drift_psi{feature="age"}[1h]) — recent drift trend।

৬ · Alert rules

YAML · ml_alerts.yml

groups:
  - name: ml-alerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
          / sum(rate(http_requests_total[5m])) > 0.01
        for: 5m
        labels: { severity: page }
        annotations:
          summary: "ML service error rate > 1%"

      - alert: HighLatency
        expr: |
          histogram_quantile(0.99,
            rate(ml_inference_latency_seconds_bucket[5m])
          ) > 0.2
        for: 10m
        labels: { severity: warn }

      - alert: DriftDetected
        expr: ml_data_drift_psi > 0.25
        for: 1h
        labels: { severity: warn }
        annotations:
          summary: "Feature {{ $labels.feature }} PSI = {{ $value }}"

      - alert: PredictionDistributionShift
        expr: |
          abs(
            sum(rate(ml_predictions_total{label="positive"}[1h]))
            / sum(rate(ml_predictions_total[1h]))
            - 0.3
          ) > 0.1
        for: 30m
        labels: { severity: page }

ML-specific alerts: error/latency standard, plus drift, plus prediction distribution shift। for: 5m — sustained, not flap।

৭ · Grafana dashboards

Service overview: RPS, error rate, p99 latency।
Model performance: accuracy time-series, confidence distribution।
Drift dashboard: per-feature PSI heatmap।
Business KPI: conversion rate (separate metric pipeline)।

Grafana JSON-based dashboards — version-control git-এ। Reusable templates।

৮ · Cardinality care

Each unique label combination = separate metric। High-cardinality labels (user_id) = explosion।

OK: model_version (~5), label (~10)।
Bad: user_id (millions), full URL paths।
Bound cardinality — bucket high-card values।

Prometheus + Grafana standard ML monitoring stack — পাঠ ২৪-২৬-এর concept-গুলো এখানে operationalized।

Bangladesh tech-এ Prometheus + Grafana de facto। Datadog/New Relic enterprise option, কিন্তু cost বেশি।

ভাবনার প্রশ্ন

প্র ০১"Prometheus vs Datadog/CloudWatch — Bangladesh team-এর জন্য?"

Cost vs convenience tradeoff।

Prometheus + Grafana (OSS):

Free; self-host।
Powerful PromQL।
Wide K8s integration।
Cons: ops overhead, retention storage।

Datadog (SaaS):

Easy setup, polished UI।
APM + log + metric one platform।
Cost: $15-30/host/month rapidly accumulates।
BD: large fintech use; smaller skip।

CloudWatch (AWS):

AWS-native, integrated।
Cost: per-metric per-dashboard charge।
Custom metric expensive at scale।

BD adoption:

~80% Prometheus + Grafana।
~10% Datadog (enterprise)।
~5% CloudWatch (AWS-locked)।

মূল উপলব্ধি: Prometheus + Grafana cost-effective default। Managed alternatives convenience cost-justified at large scale।

প্র ০২"Cardinality bombs — কীভাবে avoid?"

Cardinality issue Prometheus practical disaster।

Common bombs:

user_id label — millions।
request_path full URL — combinatorial।
request_id, session_id — unique each।
high-precision timestamps as labels।

Symptoms:

Prometheus memory explosion।
Slow queries।
Disk fill।

Prevention:

Bucket high-card values: user_id_bucket = id % 100।
Path normalize: /api/users/{id} → /api/users/_।
Allow-list labels in CI lint।
Cardinality monitor: count by (__name__) ({__name__=~".*"})।

Detection:

Prometheus's tsdb-status page — top metrics by series।
Alert: series count growing fast।

Bangladesh real:

One BD startup — request_path label exploded ৫০০K series, Prometheus crash।
Fix: relabel + drop high-cardinality।

মূল উপলব্ধি: Cardinality awareness essential। Label design critical। Quarterly review series count।

প্র ০৩"Retention strategy — কত দিন keep?"

Storage cost vs historical value।

Default Prometheus:

15 days local storage।
Sufficient most operational debugging।

Long-term storage:

Thanos, Cortex, VictoriaMetrics — Prometheus + S3 backing।
Year-long retention possible।
Cost: tiered storage (hot/cold)।

What needs long retention:

Compliance audit (Bangladesh Bank ICT 5 years)।
Long-term trend (year-over-year)।
Capacity planning।

Tiered:

15 days full granularity।
3 months downsampled (5-min average)।
1 year downsampled (1-hour average)।
Storage 100× less।

BD context:

Most teams 30-90 days local।
Critical metrics export to BigQuery/Snowflake।
Compliance separate audit trail।

মূল উপলব্ধি: Tiered retention smart cost. Downsample aggressively. Compliance separate from operational।

প্র ০৪"SLO/SLI design ML service-এ?"

SLI = Service Level Indicator (metric)। SLO = target। SLA = contract।

SLI examples ML:

Availability: % requests successful (200 status)।
Latency: % requests p99 < threshold।
Quality: % predictions confidence > threshold।
Freshness: % requests using model less than X days old।

SLO target:

99.9% availability — 43 min downtime/month।
99% latency < 100ms।
Error budget: 100% - SLO = max budget acceptable।

Error budget concept:

SLO 99.9% → 0.1% budget = 43 min/month।
If month consumed, freeze deployments।
Below budget: ship freely।
Self-regulating tension between feature + reliability।

BD context — bKash payment:

SLO 99.95% extreme reliability।
4 min/month budget tight।
ML model service must align।

SLA careful:

SLO internal target। SLA contract with customer (stricter or looser)।
Penalty risk; underpromise overdeliver।

মূল উপলব্ধি: SLO/SLI/SLA discipline ML team graduate stage। Error budget reliability + speed balance।

অনুশীলন

Instrument: উপরের code নিজের FastAPI ML service-এ apply। /metrics দেখুন।
pip install prometheus-fastapi-instrumentator; instrument; localhost:8000/metrics check।
Local Prometheus: Docker-এ Prometheus run; service scrape; Grafana add data source।
Compose with Prometheus + Grafana; prometheus.yml-এ static_configs target add; Grafana DS Prometheus add।
চিন্তা: Pathao ride matching — ৫টি SLI define করুন।
- Availability: % match request 200।
- Latency: % p99 < 200ms।
- Match quality: % rides with confidence > 0.7।
- Freshness: % decisions using model < 24h old।
- Fairness: subgroup acceptance gap < 5%।