পাঠ ৮ · ৩৩-এর মধ্যে · মডিউল ২

Home / AI Courses / MLOps / MLflow

Experiment tracking — MLflow

MLflow — tracking, registry, projects, models

৮ মিনিট পড়া মধ্য · Intermediate Python

এই পাঠে যা শিখবেন

MLflow-এর ৪টি component — কী, কেন
Tracking API — manual ও autolog
Run organize — experiments, parent/child run, tags
Production-grade backend setup

১ · কেন experiment tracking

একটি data scientist দিনে ১০-৩০টি experiment run করতে পারেন — different hyperparameters, different features. কোন combination best — মনে রাখা impossible. তাই tracking দরকার।

Without tracking: "৩ মাস আগে ৯২% পেয়েছিলাম, কিন্তু কী config-এ মনে নেই।"
With tracking: "এই run-এ AUC ০.৯২, hyperparams এই, code git SHA এই।"

২ · MLflow components

৪ component

Tracking: log params/metrics/artifacts/models per run.
Projects: reusable, reproducible code package (MLproject file).
Models: framework-agnostic model packaging format (flavors).
Model Registry: centralized versioned model lifecycle (Lesson 11).

৩ · Tracking — manual API

Python · MLflow tracking

import mlflow
import mlflow.sklearn
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score

mlflow.set_tracking_uri("http://mlflow.local:5000")
mlflow.set_experiment("bangla-sentiment")

with mlflow.start_run(run_name="rf-baseline") as run:
    # ১. params log
    n_estimators = 100
    max_depth = 8
    mlflow.log_params({
        "n_estimators": n_estimators,
        "max_depth": max_depth,
        "data_version": "v3",
    })

    # ২. train
    model = RandomForestClassifier(
        n_estimators=n_estimators, max_depth=max_depth, random_state=42
    )
    model.fit(X_train, y_train)

    # ৩. metric log
    auc = roc_auc_score(y_test, model.predict_proba(X_test)[:, 1])
    mlflow.log_metric("auc", auc)

    # ৪. model log (with signature for serving)
    mlflow.sklearn.log_model(
        model,
        artifact_path="model",
        registered_model_name="bangla-sentiment-rf",
    )

    # ৫. tag — searchable
    mlflow.set_tag("team", "nlp")
    mlflow.set_tag("env", "dev")

    print(f"Run id: {run.info.run_id}")
    print(f"AUC: {auc:.4f}")

start_run context-এ সব log নিজেই grouped. UI-তে গিয়ে এই run compare করা যাবে অন্যদের সাথে। registered_model_name দিলে — registry-তে auto-register.

৪ · Autolog — এক লাইনে

Python · MLflow autolog

import mlflow

mlflow.set_experiment("bangla-sentiment")
mlflow.autolog()  # ⭐ ১ লাইন — sklearn, PyTorch, TF, XGBoost সব track

# নিচের code কিছু change ছাড়াই — সব auto-tracked
from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier(n_estimators=100, max_depth=8)
model.fit(X_train, y_train)
preds = model.predict(X_test)

Autolog frameworks recognize করে — params, metrics, training curve, model — সব auto. Custom metric-এর জন্য তবু manual log_metric. ৯৫% case-এ autolog যথেষ্ট।

৫ · Run organization

Experiment: related runs-এর container. "fraud-v2", "bangla-sentiment".
Run: single training execution.
Parent/child run: nested. Hyperparameter sweep — parent run, প্রতি combination — child run.
Tags: arbitrary key-value — team, env, model_type. Searchable.

৬ · Production-grade setup

Default — local file. Multi-team-এ proper backend দরকার:

Backend store: Postgres / MySQL — runs, params, metrics.
Artifact store: S3 / GCS / Azure Blob / MinIO — model files, plots.
Tracking server: central HTTP server. Run as Docker container.
Authentication: reverse proxy (nginx) + OAuth, বা MLflow 2.x basic auth.

YAML · Docker compose for MLflow

version: "3"
services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_USER: mlflow
      POSTGRES_PASSWORD: mlflow
      POSTGRES_DB: mlflow
    volumes: ["pgdata:/var/lib/postgresql/data"]

  minio:
    image: minio/minio
    command: server /data --console-address ":9001"
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: minio12345
    ports: ["9000:9000", "9001:9001"]
    volumes: ["miniodata:/data"]

  mlflow:
    image: ghcr.io/mlflow/mlflow:v2.10.0
    depends_on: [postgres, minio]
    environment:
      MLFLOW_S3_ENDPOINT_URL: http://minio:9000
      AWS_ACCESS_KEY_ID: minio
      AWS_SECRET_ACCESS_KEY: minio12345
    command: >
      mlflow server
      --backend-store-uri postgresql+psycopg2://mlflow:mlflow@postgres/mlflow
      --default-artifact-root s3://mlflow/
      --host 0.0.0.0 --port 5000
    ports: ["5000:5000"]

volumes:
  pgdata:
  miniodata:

Local development-এ এই compose stack পুরো MLflow + S3-compatible storage দেয়। MinIO bucket "mlflow" আগে create করতে হবে। Production-এ — managed Postgres + cloud S3.

MLflow tracking server two backend connect করে — structured metadata-এর জন্য Postgres, file artifact-এর জন্য S3-compatible storage.

Bangladesh-এ small team-এর জন্য — sqlite backend + local artifact (single VM) — আগে ৬ মাস ভাল কাজ করে। Multi-team হলে — Postgres + S3-এ migration easy. Lock-in concern নেই।

ভাবনার প্রশ্ন

প্র ০১ "MLflow vs Weights & Biases (W&B) — কোনটা কখন?" Decision factors.

এই দু'টোই dominant experiment tracking tool — overlap বেশি, কিন্তু philosophy ভিন্ন।

MLflow strengths:

Open-source, self-hostable. Data on-prem.
Model registry built-in.
Framework agnostic + many "flavors".
Databricks-এর ecosystem-এ deeply integrated.

W&B strengths:

Polished UI, real-time charts, smooth UX.
Sweeps (hyperparameter search) built-in.
Reports — collaborative documents.
Faster onboarding — managed SaaS.

Decision factors:

Data privacy: on-prem mandate → MLflow.
SaaS preferred: small team, fast start → W&B.
Cost: MLflow free OSS; W&B paid for team plan.
Sweeps focus: heavy hyperparameter search → W&B integrated; MLflow + Optuna combo.
Bangladesh banking/regulated: on-prem MLflow + Postgres + on-prem MinIO.
NLP research team: W&B's model versioning + reports → favored.

Cost comparison (rough):

MLflow self-hosted: ~$৫০-১০০/month infrastructure.
W&B Team plan: $২০/seat/month + storage costs.
For ৫-engineer team: MLflow $১০০ vs W&B $১,২০০+ per year roughly.

Hybrid pattern:

Some teams use both — W&B for research, MLflow for production registry.
OS layer flexibility allows this.

Other contenders (২০২৫):

Neptune.ai: Polish team alternative — strong UI.
Comet ML: similar SaaS.
Aim: open-source W&B alternative.
ClearML: open-source + integrated orchestration.

মূল উপলব্ধি: MLflow vs W&B — false dichotomy in many cases. Bangladesh tech context-এ MLflow majority appropriate (cost + on-prem + open-source). Research-heavy academic team-এ W&B better UX often justifies.

প্র ০২ "১০,০০০ run after 6 months — how to query/find what we need?"

Tracking adoption-এর paradox — সফল হলে runs explode. Search UX critical.

Search/filter approaches:

(১) Tags strategically:

Every run-এ team, env, project, owner tag.
UI search: tags.team = "fraud" AND tags.env = "prod-candidate".
Programmatic: mlflow.search_runs(filter_string="...")।

(২) Experiments hierarchy:

Each project = experiment (e.g., bangla-sentiment-2025-q1).
Don't dump everything in "Default".
Naming convention: {team}-{project}-{quarter}।

(৩) Run name + run_id:

Auto-generated names ("painted-fish-37") — confusing.
Custom: mlflow.start_run(run_name="rf-100est-d8-cv5").
Self-documenting.

(৪) Parameter & metric filters:

UI-এ "metrics.auc > 0.9" filter.
Combined: best AUC + matching tags.

(৫) Compare view:

Select multiple runs → side-by-side parameter & metric table.
Parallel coordinates plot — high-dim hyperparameter visualization.

(৬) Programmatic queries:

Pandas DataFrame: mlflow.search_runs(experiment_ids=["1"], filter_string="metrics.auc > 0.9").
Top-N best runs: sort + head.
Export to BI tool (Metabase, Superset).

Cleanup strategies:

Failed/early-terminated runs: mlflow gc.
Old experiment archive: mlflow experiments delete.
Artifact retention policy — old runs-এর artifact S3 lifecycle-এ delete.

"Best run" selection patterns:

Single metric: order_by=["metrics.auc DESC"] + top-1.
Multi-metric: business metric primary, others guardrail.
"Best stable" — top-10 by mean across seeds.

Common anti-patterns:

"Default" experiment-এ everything dump.
No tag discipline.
Auto-generated run names without context.
Failed run-গুলো-ও registry-তে register.

Bangladesh team experience:

Initial enthusiasm-এ MLflow adopt → ৬ মাস পরে noisy → engineer give up.
Solution: MLflow হলেও — discipline-এর enforcement (tag + naming) critical.

মূল উপলব্ধি: Tracking sustained adoption = tooling + discipline সমান। Tag taxonomy day-1-থেকে define. Naming convention enforce in CI. ১০,০০০ runs-এ লুকানো golden run — disciplined search-এ ১ মিনিটে; chaotic store-এ ৩ ঘণ্টা।

প্র ০৩ "Backend store — sqlite vs Postgres — কখন migrate?"

MLflow backend default sqlite — single-file, file-based. Production multi-user-এ insufficient.

sqlite-এর limit:

Single writer at a time — concurrent runs serialize.
10K-100K runs পর slow.
Network access কঠিন (file lock issues over NFS).
No replication, no backup story.

Postgres benefits:

Concurrent write — multiple training jobs simultaneously.
Million+ runs scale.
Network-accessible — multiple worker nodes connect.
Standard backup, replication, monitoring.

When to migrate:

৩+ data scientist concurrent training.
৫,০০০+ accumulated runs.
Multi-host setup (training on cluster, server elsewhere).
Production registry (model promote, audit).

Migration process:

MLflow-এ built-in mlflow db upgrade.
sqlite → Postgres dump-and-load tools available.
Downtime: ১৫ মিনিট to few hours depending on size.
Test in staging first.

Practical tips:

Postgres version pin (15 stable choice ২০২৫)।
Connection pool config (PgBouncer for many concurrent runs).
Indexes — MLflow-এর schema-এ default OK; custom queries-এ tune.
Backup: nightly pg_dump + retain ৩০ দিন।

Artifact store separately migrate:

local FS → S3 — independent of backend migration.
Existing artifacts: rsync or AWS CLI.
S3 lifecycle policy: old artifacts archive (Glacier) cost reduce.

Self-hosted Postgres vs managed (RDS, Cloud SQL):

Managed: backup, patching, HA — automatic; cost ৩-৪× self-hosted.
Self-hosted: cheap; ops overhead.
Bangladesh-এ moderate-size — managed comfort worth.

"Don't migrate too late":

৫০K+ runs sqlite-এ বসে থাকলে — migration painful.
Schema upgrade deadlock-এর সুযোগ।
Better — early migration when team still small.

মূল উপলব্ধি: sqlite ১ engineer-এর জন্য fine. ২+ engineer concurrent — Postgres migrate. Migration cheap + reversible — too-early decision-এর harm কম, too-late-এর কষ্ট বেশি।

প্র ০৪ "MLflow tracking culture establish করা" — team-এ adopt করানোর strategy কী?

Tool ছাড়া adoption — culture. MLflow install easy; consistent use কঠিন।

Adoption barriers:

"Notebook-এ print যথেষ্ট" mindset.
"আমার experiment small, log করার লাগবে না"।
Setup friction — server URL, credentials, environment variable.
"আমি code change চাই না"।

Strategy ১ — Reduce friction:

MLflow URL — environment variable global, one-time setup.
Template script provided — copy-paste mlflow boilerplate.
Autolog — minimum code change.
Onboarding doc with screenshots.

Strategy ২ — Demonstrate value:

"Demo day" — best run-গুলো MLflow UI-তে show.
Comparison plot — visually impressive.
Real story: "এই run-এ ১৫% improvement, কিন্তু কেউ track করেনি — হারিয়ে গেছে।"

Strategy ৩ — Mandate gradually:

Phase 1: optional, encourage.
Phase 2: production model registry-তে যেতে — MLflow run-id required.
Phase 3: code review checklist-এ — "tracking যোগ?"
Pure mandate without education — backlash.

Strategy ৪ — Champion-driven:

একজন senior DS adopt → অন্যরা follow.
Success story share — internal Slack channel.
"এই team-এর tracking discipline best" — public recognition.

Strategy ৫ — Tag taxonomy enforce:

CI-এ check: every run must have team, project tag.
Pre-commit hook: untracked notebook reject.
Production registry promotion — only tracked runs.

Anti-patterns:

"Everyone must use MLflow from tomorrow" — top-down mandate, no support.
MLflow installed but no admin — broken often, trust eroded.
Free-for-all tagging — eventual chaos.

Bangladesh-specific note:

Junior engineers tend new tool eager-adopt; senior wary. Senior buy-in important.
Bangla docs ও internal demo — accelerate.
Local meetup-এ "MLflow journey" share — recruiting + adoption network.

Measurement:

Weekly metric: % production model traceable to MLflow run.
Goal: ৯৫%+ in ৬ months.
Outliers — investigate & support.

মূল উপলব্ধি: MLflow adoption tool-purchase না — habit-formation exercise. Friction কমান, value demonstrate, mandate gradually, champion empower. ৩-৬ months consistent effort — ১ বছর পরে "MLflow ছাড়া আমরা কীভাবে বেঁচে ছিলাম?" — দলের voice.

অনুশীলন

Setup: উপরের docker-compose.yml চালান। MLflow UI (localhost:5000) browser-এ open করুন।

docker compose up -d. MinIO console (localhost:9001) login → bucket "mlflow" create. MLflow UI-তে empty experiment দেখা যাবে।
Track: উপরের autolog code আপনার নিজের একটি sklearn project-এ apply করুন। UI-তে run আসছে?

Run finish-এর পর UI refresh — params, metrics, model artifact সব দেখা যাবে। Compare 2 runs — চমৎকার diff view.

Search: Programmatic query — top-5 highest AUC run.

import mlflow
runs = mlflow.search_runs(
    experiment_names=["bangla-sentiment"],
    order_by=["metrics.auc DESC"],
    max_results=5,
)
print(runs[["run_id", "metrics.auc", "params.n_estimators"]])