পাঠ ৮ · ৩৩-এর মধ্যে · মডিউল ২
Home / AI Courses / MLOps / MLflow

Experiment tracking — MLflow

MLflow — tracking, registry, projects, models
৮ মিনিট পড়া মধ্য · Intermediate Python

এই পাঠে যা শিখবেন

  • MLflow-এর ৪টি component — কী, কেন
  • Tracking API — manual ও autolog
  • Run organize — experiments, parent/child run, tags
  • Production-grade backend setup

১ · কেন experiment tracking

একটি data scientist দিনে ১০-৩০টি experiment run করতে পারেন — different hyperparameters, different features। কোন combination best — মনে রাখা impossible। তাই tracking দরকার।

  • Without tracking: "৩ মাস আগে ৯২% পেয়েছিলাম, কিন্তু কী config-এ মনে নেই।"
  • With tracking: "এই run-এ AUC ০.৯২, hyperparams এই, code git SHA এই।"

২ · MLflow components

৪ component

Tracking: log params/metrics/artifacts/models per run।
Projects: reusable, reproducible code package (MLproject file)।
Models: framework-agnostic model packaging format (flavors)।
Model Registry: centralized versioned model lifecycle (Lesson 11)।

৩ · Tracking — manual API

Python · MLflow tracking 
import mlflow
import mlflow.sklearn
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score

mlflow.set_tracking_uri("http://mlflow.local:5000")
mlflow.set_experiment("bangla-sentiment")

with mlflow.start_run(run_name="rf-baseline") as run:
    # ১. params log
    n_estimators = 100
    max_depth = 8
    mlflow.log_params({
        "n_estimators": n_estimators,
        "max_depth": max_depth,
        "data_version": "v3",
    })

    # ২. train
    model = RandomForestClassifier(
        n_estimators=n_estimators, max_depth=max_depth, random_state=42
    )
    model.fit(X_train, y_train)

    # ৩. metric log
    auc = roc_auc_score(y_test, model.predict_proba(X_test)[:, 1])
    mlflow.log_metric("auc", auc)

    # ৪. model log (with signature for serving)
    mlflow.sklearn.log_model(
        model,
        artifact_path="model",
        registered_model_name="bangla-sentiment-rf",
    )

    # ৫. tag — searchable
    mlflow.set_tag("team", "nlp")
    mlflow.set_tag("env", "dev")

    print(f"Run id: {run.info.run_id}")
    print(f"AUC: {auc:.4f}")

    


    

      start_run context-এ সব log নিজেই grouped। UI-তে গিয়ে এই run compare করা যাবে অন্যদের সাথে। registered_model_name দিলে — registry-তে auto-register।
    


    
৪ · Autolog — এক লাইনে


    

      

        
        Python · MLflow autolog
      

      
import mlflow

mlflow.set_experiment("bangla-sentiment")
mlflow.autolog()  # ⭐ ১ লাইন — sklearn, PyTorch, TF, XGBoost সব track

# নিচের code কিছু change ছাড়াই — সব auto-tracked
from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier(n_estimators=100, max_depth=8)
model.fit(X_train, y_train)
preds = model.predict(X_test)

      

        
        
      

      

    


    

      Autolog frameworks recognize করে — params, metrics, training curve, model — সব auto। Custom metric-এর জন্য তবু manual log_metric। ৯৫% case-এ autolog যথেষ্ট।
    


    
৫ · Run organization


    
    
      
  • Experiment: related runs-এর container। "fraud-v2", "bangla-sentiment"।
    • 
      
  • Run: single training execution।
    • 
      
  • Parent/child run: nested। Hyperparameter sweep — parent run, প্রতি combination — child run।
    • 
      
  • Tags: arbitrary key-value — team, env, model_type। Searchable।
    • 
    


    
৬ · Production-grade setup


    

      Default — local file। Multi-team-এ proper backend দরকার:
    

    
    
      
  • Backend store: Postgres / MySQL — runs, params, metrics।
    • 
      
  • Artifact store: S3 / GCS / Azure Blob / MinIO — model files, plots।
    • 
      
  • Tracking server: central HTTP server। Run as Docker container।
    • 
      
  • Authentication: reverse proxy (nginx) + OAuth, বা MLflow 2.x basic auth।
    • 
    


    

      

        
        YAML · Docker compose for MLflow
      

      
version: "3"
services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_USER: mlflow
      POSTGRES_PASSWORD: mlflow
      POSTGRES_DB: mlflow
    volumes: ["pgdata:/var/lib/postgresql/data"]

  minio:
    image: minio/minio
    command: server /data --console-address ":9001"
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: minio12345
    ports: ["9000:9000", "9001:9001"]
    volumes: ["miniodata:/data"]

  mlflow:
    image: ghcr.io/mlflow/mlflow:v2.10.0
    depends_on: [postgres, minio]
    environment:
      MLFLOW_S3_ENDPOINT_URL: http://minio:9000
      AWS_ACCESS_KEY_ID: minio
      AWS_SECRET_ACCESS_KEY: minio12345
    command: >
      mlflow server
      --backend-store-uri postgresql+psycopg2://mlflow:mlflow@postgres/mlflow
      --default-artifact-root s3://mlflow/
      --host 0.0.0.0 --port 5000
    ports: ["5000:5000"]

volumes:
  pgdata:
  miniodata:

      

        
        
      

      

    


    

      Local development-এ এই compose stack পুরো MLflow + S3-compatible storage দেয়। MinIO bucket "mlflow" আগে create করতে হবে। Production-এ — managed Postgres + cloud S3।
    


    

      
        
          
            
          
        

        MLflow architecture — 3-tier
        Client → Tracking server → Backend + Artifact stores

        
        
        Training script
        mlflow.log_*()
        mlflow.autolog()

        
        
        MLflow server
        REST API + UI
        port 5000

        
        
        Backend store
        Postgres
        runs, params, metrics

        
        
        Artifact store
        S3 / GCS / MinIO
        model files, plots

        
        
        Browser UI
        compare runs, charts

        
        
        
        
        

        Backend = structured data; Artifact = files; Server = orchestrator।
      
      
MLflow tracking server two backend connect করে — structured metadata-এর জন্য Postgres, file artifact-এর জন্য S3-compatible storage।

    


    

      Bangladesh-এ small team-এর জন্য — sqlite backend + local artifact (single VM) — আগে ৬ মাস ভাল কাজ করে। Multi-team হলে — Postgres + S3-এ migration easy। Lock-in concern নেই।
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      
        প্র ০১
        "MLflow vs Weights & Biases (W&B) — কোনটা কখন?" Decision factors।
      
      

        

          এই দু'টোই dominant experiment tracking tool — overlap বেশি, কিন্তু philosophy ভিন্ন।
        

        
MLflow strengths:

        
    
          
  • Open-source, self-hostable। Data on-prem।
    • 
          
  • Model registry built-in।
    • 
          
  • Framework agnostic + many "flavors"।
    • 
          
  • Databricks-এর ecosystem-এ deeply integrated।
    • 
        

        
W&B strengths:

        
    
          
  • Polished UI, real-time charts, smooth UX।
    • 
          
  • Sweeps (hyperparameter search) built-in।
    • 
          
  • Reports — collaborative documents।
    • 
          
  • Faster onboarding — managed SaaS।
    • 
        

        
Decision factors:

        
    
          
  • Data privacy: on-prem mandate → MLflow।
    • 
          
  • SaaS preferred: small team, fast start → W&B।
    • 
          
  • Cost: MLflow free OSS; W&B paid for team plan।
    • 
          
  • Sweeps focus: heavy hyperparameter search → W&B integrated; MLflow + Optuna combo।
    • 
          
  • Bangladesh banking/regulated: on-prem MLflow + Postgres + on-prem MinIO।
    • 
          
  • NLP research team: W&B's model versioning + reports → favored।
    • 
        

        
Cost comparison (rough):

        
    
          
  • MLflow self-hosted: ~$৫০-১০০/month infrastructure।
    • 
          
  • W&B Team plan: $২০/seat/month + storage costs।
    • 
          
  • For ৫-engineer team: MLflow $১০০ vs W&B $১,২০০+ per year roughly।
    • 
        

        
Hybrid pattern:

        
    
          
  • Some teams use both — W&B for research, MLflow for production registry।
    • 
          
  • OS layer flexibility allows this।
    • 
        

        
Other contenders (২০২৫):

        
    
          
  • Neptune.ai: Polish team alternative — strong UI।
    • 
          
  • Comet ML: similar SaaS।
    • 
          
  • Aim: open-source W&B alternative।
    • 
          
  • ClearML: open-source + integrated orchestration।
    • 
        

        

          মূল উপলব্ধি: MLflow vs W&B — false dichotomy in many cases। Bangladesh tech context-এ MLflow majority appropriate (cost + on-prem + open-source)। Research-heavy academic team-এ W&B better UX often justifies।
        

      

    


    

      
        প্র ০২
        "১০,০০০ run after 6 months — how to query/find what we need?"
      
      

        

          Tracking adoption-এর paradox — সফল হলে runs explode। Search UX critical।
        

        
Search/filter approaches:

        
(১) Tags strategically:

        
    
          
  • Every run-এ team, env, project, owner tag।
    • 
          
  • UI search: tags.team = "fraud" AND tags.env = "prod-candidate"।
    • 
          
  • Programmatic: mlflow.search_runs(filter_string="...")।
    • 
        

        
(২) Experiments hierarchy:

        
    
          
  • Each project = experiment (e.g., bangla-sentiment-2025-q1)।
    • 
          
  • Don't dump everything in "Default"।
    • 
          
  • Naming convention: {team}-{project}-{quarter}।
    • 
        

        
(৩) Run name + run_id:

        
    
          
  • Auto-generated names ("painted-fish-37") — confusing।
    • 
          
  • Custom: mlflow.start_run(run_name="rf-100est-d8-cv5")।
    • 
          
  • Self-documenting।
    • 
        

        
(৪) Parameter & metric filters:

        
    
          
  • UI-এ "metrics.auc > 0.9" filter।
    • 
          
  • Combined: best AUC + matching tags।
    • 
        

        
(৫) Compare view:

        
    
          
  • Select multiple runs → side-by-side parameter & metric table।
    • 
          
  • Parallel coordinates plot — high-dim hyperparameter visualization।
    • 
        

        
(৬) Programmatic queries:

        
    
          
  • Pandas DataFrame: mlflow.search_runs(experiment_ids=["1"], filter_string="metrics.auc > 0.9")।
    • 
          
  • Top-N best runs: sort + head।
    • 
          
  • Export to BI tool (Metabase, Superset)।
    • 
        

        
Cleanup strategies:

        
    
          
  • Failed/early-terminated runs: mlflow gc।
    • 
          
  • Old experiment archive: mlflow experiments delete।
    • 
          
  • Artifact retention policy — old runs-এর artifact S3 lifecycle-এ delete।
    • 
        

        
"Best run" selection patterns:

        
    
          
  • Single metric: order_by=["metrics.auc DESC"] + top-1।
    • 
          
  • Multi-metric: business metric primary, others guardrail।
    • 
          
  • "Best stable" — top-10 by mean across seeds।
    • 
        

        
Common anti-patterns:

        
    
          
  • "Default" experiment-এ everything dump।
    • 
          
  • No tag discipline।
    • 
          
  • Auto-generated run names without context।
    • 
          
  • Failed run-গুলো-ও registry-তে register।
    • 
        

        
Bangladesh team experience:

        
    
          
  • Initial enthusiasm-এ MLflow adopt → ৬ মাস পরে noisy → engineer give up।
    • 
          
  • Solution: MLflow হলেও — discipline-এর enforcement (tag + naming) critical।
    • 
        

        

          মূল উপলব্ধি: Tracking sustained adoption = tooling + discipline সমান। Tag taxonomy day-1-থেকে define। Naming convention enforce in CI। ১০,০০০ runs-এ লুকানো golden run — disciplined search-এ ১ মিনিটে; chaotic store-এ ৩ ঘণ্টা।
        

      

    


    

      
        প্র ০৩
        "Backend store — sqlite vs Postgres — কখন migrate?"
      
      

        

          MLflow backend default sqlite — single-file, file-based। Production multi-user-এ insufficient।
        

        
sqlite-এর limit:

        
    
          
  • Single writer at a time — concurrent runs serialize।
    • 
          
  • 10K-100K runs পর slow।
    • 
          
  • Network access কঠিন (file lock issues over NFS)।
    • 
          
  • No replication, no backup story।
    • 
        

        
Postgres benefits:

        
    
          
  • Concurrent write — multiple training jobs simultaneously।
    • 
          
  • Million+ runs scale।
    • 
          
  • Network-accessible — multiple worker nodes connect।
    • 
          
  • Standard backup, replication, monitoring।
    • 
        

        
When to migrate:

        
    
          
  • ৩+ data scientist concurrent training।
    • 
          
  • ৫,০০০+ accumulated runs।
    • 
          
  • Multi-host setup (training on cluster, server elsewhere)।
    • 
          
  • Production registry (model promote, audit)।
    • 
        

        
Migration process:

        
    
          
  • MLflow-এ built-in mlflow db upgrade।
    • 
          
  • sqlite → Postgres dump-and-load tools available।
    • 
          
  • Downtime: ১৫ মিনিট to few hours depending on size।
    • 
          
  • Test in staging first।
    • 
        

        
Practical tips:

        
    
          
  • Postgres version pin (15 stable choice ২০২৫)।
    • 
          
  • Connection pool config (PgBouncer for many concurrent runs)।
    • 
          
  • Indexes — MLflow-এর schema-এ default OK; custom queries-এ tune।
    • 
          
  • Backup: nightly pg_dump + retain ৩০ দিন।
    • 
        

        
Artifact store separately migrate:

        
    
          
  • local FS → S3 — independent of backend migration।
    • 
          
  • Existing artifacts: rsync or AWS CLI।
    • 
          
  • S3 lifecycle policy: old artifacts archive (Glacier) cost reduce।
    • 
        

        
Self-hosted Postgres vs managed (RDS, Cloud SQL):

        
    
          
  • Managed: backup, patching, HA — automatic; cost ৩-৪× self-hosted।
    • 
          
  • Self-hosted: cheap; ops overhead।
    • 
          
  • Bangladesh-এ moderate-size — managed comfort worth।
    • 
        

        
"Don't migrate too late":

        
    
          
  • ৫০K+ runs sqlite-এ বসে থাকলে — migration painful।
    • 
          
  • Schema upgrade deadlock-এর সুযোগ।
    • 
          
  • Better — early migration when team still small।
    • 
        

        

          মূল উপলব্ধি: sqlite ১ engineer-এর জন্য fine। ২+ engineer concurrent — Postgres migrate। Migration cheap + reversible — too-early decision-এর harm কম, too-late-এর কষ্ট বেশি।
        

      

    


    

      
        প্র ০৪
        "MLflow tracking culture establish করা" — team-এ adopt করানোর strategy কী?
      
      

        

          Tool ছাড়া adoption — culture। MLflow install easy; consistent use কঠিন।
        

        
Adoption barriers:

        
    
          
  • "Notebook-এ print যথেষ্ট" mindset।
    • 
          
  • "আমার experiment small, log করার লাগবে না"।
    • 
          
  • Setup friction — server URL, credentials, environment variable।
    • 
          
  • "আমি code change চাই না"।
    • 
        

        
Strategy ১ — Reduce friction:

        
    
          
  • MLflow URL — environment variable global, one-time setup।
    • 
          
  • Template script provided — copy-paste mlflow boilerplate।
    • 
          
  • Autolog — minimum code change।
    • 
          
  • Onboarding doc with screenshots।
    • 
        

        
Strategy ২ — Demonstrate value:

        
    
          
  • "Demo day" — best run-গুলো MLflow UI-তে show।
    • 
          
  • Comparison plot — visually impressive।
    • 
          
  • Real story: "এই run-এ ১৫% improvement, কিন্তু কেউ track করেনি — হারিয়ে গেছে।"
    • 
        

        
Strategy ৩ — Mandate gradually:

        
    
          
  • Phase 1: optional, encourage।
    • 
          
  • Phase 2: production model registry-তে যেতে — MLflow run-id required।
    • 
          
  • Phase 3: code review checklist-এ — "tracking যোগ?"
    • 
          
  • Pure mandate without education — backlash।
    • 
        

        
Strategy ৪ — Champion-driven:

        
    
          
  • একজন senior DS adopt → অন্যরা follow।
    • 
          
  • Success story share — internal Slack channel।
    • 
          
  • "এই team-এর tracking discipline best" — public recognition।
    • 
        

        
Strategy ৫ — Tag taxonomy enforce:

        
    
          
  • CI-এ check: every run must have team, project tag।
    • 
          
  • Pre-commit hook: untracked notebook reject।
    • 
          
  • Production registry promotion — only tracked runs।
    • 
        

        
Anti-patterns:

        
    
          
  • "Everyone must use MLflow from tomorrow" — top-down mandate, no support।
    • 
          
  • MLflow installed but no admin — broken often, trust eroded।
    • 
          
  • Free-for-all tagging — eventual chaos।
    • 
        

        
Bangladesh-specific note:

        
    
          
  • Junior engineers tend new tool eager-adopt; senior wary। Senior buy-in important।
    • 
          
  • Bangla docs ও internal demo — accelerate।
    • 
          
  • Local meetup-এ "MLflow journey" share — recruiting + adoption network।
    • 
        

        
Measurement:

        
    
          
  • Weekly metric: % production model traceable to MLflow run।
    • 
          
  • Goal: ৯৫%+ in ৬ months।
    • 
          
  • Outliers — investigate & support।
    • 
        

        

          মূল উপলব্ধি: MLflow adoption tool-purchase না — habit-formation exercise। Friction কমান, value demonstrate, mandate gradually, champion empower। ৩-৬ months consistent effort — ১ বছর পরে "MLflow ছাড়া আমরা কীভাবে বেঁচে ছিলাম?" — দলের voice।
        

      

    

  


  

    
 অনুশীলন

    
    
      
  1. 
        Setup: উপরের docker-compose.yml চালান। MLflow UI (localhost:5000) browser-এ open করুন।
        
    
          
          
    
            
    docker compose up -d। MinIO console (localhost:9001) login → bucket "mlflow" create। MLflow UI-তে empty experiment দেখা যাবে।
    
          
    
        
    
      
    2. 
      
  2. 
        Track: উপরের autolog code আপনার নিজের একটি sklearn project-এ apply করুন। UI-তে run আসছে?
        
    
          
          
    
            
    Run finish-এর পর UI refresh — params, metrics, model artifact সব দেখা যাবে। Compare 2 runs — চমৎকার diff view।
    
          
    
        
    
      
    3. 
      
  3. 
        Search: Programmatic query — top-5 highest AUC run।
        
    
          
          
    
            
    import mlflow
runs = mlflow.search_runs(
    experiment_names=["bangla-sentiment"],
    order_by=["metrics.auc DESC"],
    max_results=5,
)
print(runs[["run_id", "metrics.auc", "params.n_estimators"]])
    
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন

    
  







  
  
     এই পাঠে ভুল পেয়েছেন? জানান
  





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ৭ · Kubernetes