পাঠ ২৭ · ৩৩-এর মধ্যে · মডিউল ৪

Home / AI Courses / MLOps / Prometheus & Grafana

Prometheus ও Grafana

Prometheus + Grafana for ML monitoring

৭ মিনিট পড়া মধ্য · Intermediate Tooling

এই পাঠে যা শিখবেন

Prometheus pull model + service discovery
৪ metric types — কোনটা কখন
FastAPI-এ ML metrics instrumentation
PromQL basics + Grafana dashboard

১ · Pull-based architecture

Prometheus standard pull model — service expose HTTP /metrics; Prometheus periodically scrape (15-60 sec interval).

Pros: no metric server-overload (scrape rate Prometheus-controlled).
Pros: service-discovery friendly (K8s, Consul).
Cons: short-lived job tricky (Pushgateway alternative).

২ · Metric types

Counter: monotonically increasing — request count, error count.
Gauge: goes up/down — queue depth, model accuracy.
Histogram: distribution — latency p50/p95/p99 observable.
Summary: pre-computed quantile — like histogram but server-side.

৩ · FastAPI-এ ML metrics

Python · prometheus-fastapi-instrumentator

from fastapi import FastAPI
from prometheus_fastapi_instrumentator import Instrumentator
from prometheus_client import Counter, Histogram, Gauge

app = FastAPI()

# ML-specific custom metrics
predictions_total = Counter(
    "ml_predictions_total",
    "Total predictions made",
    ["model_version", "label"],
)
prediction_confidence = Histogram(
    "ml_prediction_confidence",
    "Confidence distribution",
    ["model_version"],
    buckets=[0.1, 0.3, 0.5, 0.7, 0.9, 1.0],
)
inference_latency = Histogram(
    "ml_inference_latency_seconds",
    "Model inference time",
    ["model_version"],
    buckets=[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0],
)
data_drift_psi = Gauge(
    "ml_data_drift_psi",
    "Population Stability Index per feature",
    ["feature"],
)

# Auto-instrument HTTP metrics
Instrumentator().instrument(app).expose(app)

@app.post("/predict")
def predict(req):
    import time
    start = time.perf_counter()
    proba = model.predict_proba([req.text])[0]
    label = str(proba.argmax())
    confidence = float(proba.max())

    elapsed = time.perf_counter() - start
    predictions_total.labels(
        model_version=MODEL_VERSION, label=label
    ).inc()
    prediction_confidence.labels(
        model_version=MODEL_VERSION
    ).observe(confidence)
    inference_latency.labels(
        model_version=MODEL_VERSION
    ).observe(elapsed)

    return {"label": label, "confidence": confidence}

# /metrics endpoint auto-exposed by Instrumentator

ML-specific metrics: prediction count by label (distribution skew), confidence histogram (drift signal), per-version latency. Drift PSI separate scrape job from data validation pipeline.

৪ · Prometheus config

YAML · prometheus.yml

global:
  scrape_interval: 30s

scrape_configs:
  - job_name: 'ml-services'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: '.*ml-.*'
        action: keep

  - job_name: 'sentiment-api'
    static_configs:
      - targets: ['sentiment-api:8000']

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

rule_files:
  - 'ml_alerts.yml'

K8s service discovery — pod label-based filter. Alertmanager-এ rules forward.

৫ · PromQL basics

rate(http_requests_total[5m]) — RPS over 5 min.
histogram_quantile(0.99, rate(ml_inference_latency_seconds_bucket[5m])) — p99 latency.
sum by (model_version) (rate(ml_predictions_total[5m])) — RPS per version.
avg_over_time(ml_data_drift_psi{feature="age"}[1h]) — recent drift trend.

৬ · Alert rules

YAML · ml_alerts.yml

groups:
  - name: ml-alerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
          / sum(rate(http_requests_total[5m])) > 0.01
        for: 5m
        labels: { severity: page }
        annotations:
          summary: "ML service error rate > 1%"

      - alert: HighLatency
        expr: |
          histogram_quantile(0.99,
            rate(ml_inference_latency_seconds_bucket[5m])
          ) > 0.2
        for: 10m
        labels: { severity: warn }

      - alert: DriftDetected
        expr: ml_data_drift_psi > 0.25
        for: 1h
        labels: { severity: warn }
        annotations:
          summary: "Feature {{ $labels.feature }} PSI = {{ $value }}"

      - alert: PredictionDistributionShift
        expr: |
          abs(
            sum(rate(ml_predictions_total{label="positive"}[1h]))
            / sum(rate(ml_predictions_total[1h]))
            - 0.3
          ) > 0.1
        for: 30m
        labels: { severity: page }

ML-specific alerts: error/latency standard, plus drift, plus prediction distribution shift. for: 5m — sustained, not flap.

৭ · Grafana dashboards

Service overview: RPS, error rate, p99 latency.
Model performance: accuracy time-series, confidence distribution.
Drift dashboard: per-feature PSI heatmap.
Business KPI: conversion rate (separate metric pipeline).

Grafana JSON-based dashboards — version-control git-এ। Reusable templates.

৮ · Cardinality care

Each unique label combination = separate metric. High-cardinality labels (user_id) = explosion.

OK: model_version (~5), label (~10).
Bad: user_id (millions), full URL paths.
Bound cardinality — bucket high-card values.

Prometheus + Grafana standard ML monitoring stack — পাঠ ২৪-২৬-এর concept-গুলো এখানে operationalized.

Bangladesh tech-এ Prometheus + Grafana de facto. Datadog/New Relic enterprise option, কিন্তু cost বেশি।

ভাবনার প্রশ্ন

প্র ০১"Prometheus vs Datadog/CloudWatch — Bangladesh team-এর জন্য?"

Cost vs convenience tradeoff.

Prometheus + Grafana (OSS):

Free; self-host.
Powerful PromQL.
Wide K8s integration.
Cons: ops overhead, retention storage.

Datadog (SaaS):

Easy setup, polished UI.
APM + log + metric one platform.
Cost: $15-30/host/month rapidly accumulates.
BD: large fintech use; smaller skip.

CloudWatch (AWS):

AWS-native, integrated.
Cost: per-metric per-dashboard charge.
Custom metric expensive at scale.

BD adoption:

~80% Prometheus + Grafana.
~10% Datadog (enterprise).
~5% CloudWatch (AWS-locked).

মূল উপলব্ধি: Prometheus + Grafana cost-effective default. Managed alternatives convenience cost-justified at large scale.

প্র ০২"Cardinality bombs — কীভাবে avoid?"

Cardinality issue Prometheus practical disaster.

Common bombs:

user_id label — millions.
request_path full URL — combinatorial.
request_id, session_id — unique each.
high-precision timestamps as labels.

Symptoms:

Prometheus memory explosion.
Slow queries.
Disk fill.

Prevention:

Bucket high-card values: user_id_bucket = id % 100.
Path normalize: /api/users/{id} → /api/users/_।
Allow-list labels in CI lint.
Cardinality monitor: count by (__name__) ({__name__=~".*"})।

Detection:

Prometheus's tsdb-status page — top metrics by series.
Alert: series count growing fast.

Bangladesh real:

One BD startup — request_path label exploded ৫০০K series, Prometheus crash.
Fix: relabel + drop high-cardinality.

মূল উপলব্ধি: Cardinality awareness essential. Label design critical. Quarterly review series count.

প্র ০৩"Retention strategy — কত দিন keep?"

Storage cost vs historical value.

Default Prometheus:

15 days local storage.
Sufficient most operational debugging.

Long-term storage:

Thanos, Cortex, VictoriaMetrics — Prometheus + S3 backing.
Year-long retention possible.
Cost: tiered storage (hot/cold).

What needs long retention:

Compliance audit (Bangladesh Bank ICT 5 years).
Long-term trend (year-over-year).
Capacity planning.

Tiered:

15 days full granularity.
3 months downsampled (5-min average).
1 year downsampled (1-hour average).
Storage 100× less.

BD context:

Most teams 30-90 days local.
Critical metrics export to BigQuery/Snowflake.
Compliance separate audit trail.

মূল উপলব্ধি: Tiered retention smart cost. Downsample aggressively. Compliance separate from operational.

প্র ০৪"SLO/SLI design ML service-এ?"

SLI = Service Level Indicator (metric). SLO = target. SLA = contract.

SLI examples ML:

Availability: % requests successful (200 status).
Latency: % requests p99 < threshold.
Quality: % predictions confidence > threshold.
Freshness: % requests using model less than X days old.

SLO target:

99.9% availability — 43 min downtime/month.
99% latency < 100ms.
Error budget: 100% - SLO = max budget acceptable.

Error budget concept:

SLO 99.9% → 0.1% budget = 43 min/month.
If month consumed, freeze deployments.
Below budget: ship freely.
Self-regulating tension between feature + reliability.

BD context — bKash payment:

SLO 99.95% extreme reliability.
4 min/month budget tight.
ML model service must align.

SLA careful:

SLO internal target. SLA contract with customer (stricter or looser).
Penalty risk; underpromise overdeliver.

মূল উপলব্ধি: SLO/SLI/SLA discipline ML team graduate stage. Error budget reliability + speed balance.

অনুশীলন

Instrument: উপরের code নিজের FastAPI ML service-এ apply. /metrics দেখুন।
pip install prometheus-fastapi-instrumentator; instrument; localhost:8000/metrics check.
Local Prometheus: Docker-এ Prometheus run; service scrape; Grafana add data source.
Compose with Prometheus + Grafana; prometheus.yml-এ static_configs target add; Grafana DS Prometheus add.
চিন্তা: Pathao ride matching — ৫টি SLI define করুন।
- Availability: % match request 200.
- Latency: % p99 < 200ms.
- Match quality: % rides with confidence > 0.7.
- Freshness: % decisions using model < 24h old.
- Fairness: subgroup acceptance gap < 5%.