পাঠ ১৩ · ৩৩-এর মধ্যে · মডিউল ২
Home / AI Courses / MLOps / Training orchestration

Training pipeline orchestration

Training orchestration — Airflow, Prefect, Dagster
৭ মিনিট পড়া উচ্চ · Advanced Python DAG

এই পাঠে যা শিখবেন

  • Workflow orchestrator কেন দরকার — bash/cron-এর সীমা
  • Airflow DAG structure — task, operator, dependency
  • Idempotent design — retry-safe
  • Tool comparison — Airflow vs Prefect vs Dagster vs Argo

১ · কেন orchestrator

একটি ML training pipeline-এ multiple step: data fetch → validate → preprocess → feature engineer → train → evaluate → register → notify। প্রত্যেক-step-এ failure possible। Cron + bash দিয়ে retry, alerting, dependency, lineage — সব manual।

  • Without orchestrator: bash script with bash chains, cron schedule। Failure-এ silent।
  • With orchestrator: DAG, retry, alert, lineage tracking, UI।

২ · DAG fundamentals

DAGDAG (Directed Acyclic Graph)edges directional, no cycle। Workflow এর mathematical model — A → B → C, কোনো backward loop না। = task-গুলোর dependency graph। Cycle না (no infinite loop)।

৩ · Airflow DAG example

Python · Airflow DAG 
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.providers.amazon.aws.operators.s3 import S3ListOperator
from datetime import datetime, timedelta

default_args = {
    "owner": "ml-team",
    "retries": 2,
    "retry_delay": timedelta(minutes=5),
    "email_on_failure": True,
}

with DAG(
    dag_id="bangla_sentiment_training",
    schedule="0 2 * * *",   # daily 2 AM
    start_date=datetime(2025, 1, 1),
    default_args=default_args,
    catchup=False,
    tags=["ml", "nlp"],
) as dag:

    def fetch_data(**ctx):
        # data fetch logic
        ds = ctx["ds"]  # execution date
        # download to s3://bucket/data/{ds}/
        return f"s3://bucket/data/{ds}/"

    def validate(**ctx):
        ti = ctx["ti"]
        path = ti.xcom_pull(task_ids="fetch_data")
        # schema, range check
        # raise if fails

    def train(**ctx):
        # train, log to MLflow, register if good
        pass

    def evaluate(**ctx):
        # evaluate against baseline, fairness gate
        pass

    def deploy(**ctx):
        # promote in registry if all gates pass
        pass

    fetch_data_task = PythonOperator(
        task_id="fetch_data", python_callable=fetch_data,
    )
    validate_task = PythonOperator(
        task_id="validate", python_callable=validate,
    )
    train_task = PythonOperator(
        task_id="train", python_callable=train,
    )
    evaluate_task = PythonOperator(
        task_id="evaluate", python_callable=evaluate,
    )
    deploy_task = PythonOperator(
        task_id="deploy", python_callable=deploy,
    )

    fetch_data_task >> validate_task >> train_task >> evaluate_task >> deploy_task

    


    

      Daily 2 AM trigger, retry 2x, failure email। UI-তে DAG visualize, individual task rerun possible।
    


    
৪ · Idempotent task design


    

      Task যে কোনো বার rerun-এ same result দেবে। Otherwise retry harmful।
    


    
    
      
  • File output: overwrite, not append। "data/{ds}/file.csv" — execution-date-keyed।
    • 
      
  • DB write: upsert, not insert।
    • 
      
  • External API: idempotency-key header।
    • 
      
  • Side effects: minimize; explicit cleanup if any।
    • 
    


    

      Anti-pattern: task-এ "append to today's log file"। Retry → duplicates। Solution: write to {ds}/output.csv — overwrite।
    


    
৫ · Tool comparison


    
    
      
  • Airflow (২০১৪, Apache):
        
      
          
    • Most mature, widest community।
      • 
          
    • Python-defined DAG, rich UI, scheduler।
      • 
          
    • Used by Airbnb, Pinterest, Uber, BD top-tech।
      • 
          
    • Drawback: heavyweight, dynamic DAG awkward।
      • 
        
    
      
    • 
      
  • Prefect (২০১৮):
        
      
          
    • "Negative engineering" focus — failure-first design।
      • 
          
    • Python-native flow, dynamic mapping easier।
      • 
          
    • Hybrid execution: cloud + local agent।
      • 
          
    • Drawback: smaller community, recent v2 breaking changes।
      • 
        
    
      
    • 
      
  • Dagster (২০১৯):
        
      
          
    • Asset-centric — outputs first-class।
      • 
          
    • Type-checked, lineage built-in।
      • 
          
    • Best for data-heavy pipelines।
      • 
          
    • Drawback: learning curve, less ML-specific।
      • 
        
    
      
    • 
      
  • Argo Workflows (K8s-native):
        
      
          
    • YAML-defined, container-per-step।
      • 
          
    • Best for K8s shop, GPU-heavy training।
      • 
          
    • Drawback: YAML verbose, less Python-friendly।
      • 
        
    
      
    • 
      
  • Kubeflow Pipelines: Lesson 14 — ML-specific Argo।
    • 
    


    
৬ · Scheduling patterns


    
    
      
  • Cron schedule: "0 2 * * *" — daily 2 AM।
    • 
      
  • Event-driven: S3 file landing → trigger।
    • 
      
  • Sensor: wait for upstream — file, DB row, API condition।
    • 
      
  • Manual: UI button — ad-hoc retraining।
    • 
      
  • Backfill: historical date range catch-up।
    • 
    


    
৭ · Long-running training jobs


    
    
      
  • Airflow worker process-এ ৬ ঘণ্টা training — বাজে। Worker tied up।
    • 
      
  • Pattern: KubernetesPodOperator — separate pod-এ training, Airflow শুধু trigger ও wait।
    • 
      
  • Bigger jobs: SparkSubmitOperator, EMR operator — external compute।
    • 
      
  • Status polling — async wait।
    • 
    


    

      
        
          
            
          
        

        ML training DAG — daily schedule
        Idempotent steps with retry + alerts

        
        Fetch
        S3 sync

        
        Validate
        schema, drift

        
        Train
        K8s GPU pod

        
        Evaluate
        vs baseline

        
        Register
        if pass gates

        
        Alert (Slack)
        on failure

        
        MLflow logging
        runs, metrics

        
        
        
        

        
        
      
      
Airflow DAG-এর task chain: success path (left to right), failure → Slack alert, training → MLflow logging side-effect।

    


    

      Bangladesh-এ Airflow most-adopted। managed (Astronomer, MWAA) cost matters — many BD teams self-host on K8s।
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      
        প্র ০১
        "Airflow vs Prefect vs Dagster — Bangladesh team choose কী criteria-তে?"
      
      

        

          Tool choice — team skill, ecosystem fit, ও specific need-এর function।
        

        
Choose Airflow when:

        
    
          
  • Existing Hadoop/Spark/HDFS ecosystem।
    • 
          
  • Team prefers established, well-documented।
    • 
          
  • Many SQL-based pipelines।
    • 
          
  • Operators-এর rich library matters (Snowflake, Redshift, etc.)।
    • 
        

        
Choose Prefect when:

        
    
          
  • Pythonic flow design preferred।
    • 
          
  • Dynamic DAG (loop over variable items) common।
    • 
          
  • Hybrid cloud + local execution।
    • 
          
  • Team comfortable with newer tool।
    • 
        

        
Choose Dagster when:

        
    
          
  • Data lineage critical।
    • 
          
  • Asset-based thinking fits team।
    • 
          
  • Type checking valued।
    • 
          
  • Modern data stack (dbt, Airbyte)।
    • 
        

        
Choose Argo when:

        
    
          
  • K8s-native shop।
    • 
          
  • GPU-heavy training, container-per-step preference।
    • 
          
  • Existing K8s expertise।
    • 
        

        
Bangladesh adoption (rough):

        
    
          
  • Airflow: 70%।
    • 
          
  • Prefect: 15%।
    • 
          
  • Dagster: 5%।
    • 
          
  • Argo + KFP: 10%।
    • 
        

        
Migration considerations:

        
    
          
  • Costly — DAG rewrite।
    • 
          
  • "Hybrid" possible — old DAGs Airflow, new Prefect।
    • 
          
  • Don't migrate without clear benefit।
    • 
        

        
Cost:

        
    
          
  • All free OSS।
    • 
          
  • Managed: MWAA $300+/month, Prefect Cloud $/user, Dagster Cloud also $।
    • 
          
  • Self-hosted: Airflow heavyweight (Postgres + scheduler + workers); Prefect simpler।
    • 
        

        

          মূল উপলব্ধি: Default Airflow for most BD teams। Prefect for newer Python-heavy team। Dagster niche-perfect for data engineering-heavy। Argo K8s-shop-এ। Choose carefully — switching cost real।
        

      

    


    

      
        প্র ০২
        "Cron vs event-driven trigger — কখন কোনটা?"
      
      

        

          Trigger choice — predictability vs reactivity trade-off।
        

        
Cron — predictable, scheduled:

        
    
          
  • Daily/hourly retraining।
    • 
          
  • Resource planning easy।
    • 
          
  • Cost predictable।
    • 
          
  • "2 AM run, finish by 4 AM" — operational rhythm।
    • 
        

        
Cron drawbacks:

        
    
          
  • Run-when-no-data — wasted compute।
    • 
          
  • Data ready at 1:30 AM — wait until 2 AM cron।
    • 
        

        
Event-driven — reactive:

        
    
          
  • S3 file landed → trigger immediately।
    • 
          
  • API webhook → trigger।
    • 
          
  • Drift alert → retraining trigger।
    • 
        

        
Event-driven drawbacks:

        
    
          
  • Burst load — many events same time।
    • 
          
  • Debugging harder — when did this trigger?
    • 
          
  • Failure mode — event missed, no run।
    • 
        

        
Hybrid common:

        
    
          
  • Daily cron-based training (default, predictable)।
    • 
          
  • Event-driven on top — drift detected → ad-hoc retrain।
    • 
          
  • Sensor pattern — DAG sit waiting until upstream data lands, then proceed।
    • 
        

        
Bangladesh examples:

        
    
          
  • bKash fraud — event-driven (transaction stream)।
    • 
          
  • Daraz recommendation — daily cron retraining।
    • 
          
  • Pathao surge — both: daily baseline + event-driven peaks।
    • 
        

        
Idempotency critical for events:

        
    
          
  • Event delivered twice — handle gracefully।
    • 
          
  • Idempotency key in payload।
    • 
          
  • Dedup table।
    • 
        

        

          মূল উপলব্ধি: Cron = train regularly, event = train when needed। Most production systems hybrid। Start cron-only; add event-trigger for specific spike scenarios।
        

      

    


    

      
        প্র ০৩
        "Long training jobs (10+ hours) Airflow-এ — proper pattern কী?"
      
      

        

          Long training Airflow worker-এ run করা = anti-pattern।
        

        
Why anti-pattern:

        
    
          
  • Worker process tied up — other DAGs blocked।
    • 
          
  • Worker restart — training lost।
    • 
          
  • Resource sharing impossible — can't right-size GPU।
    • 
        

        
Right pattern — KubernetesPodOperator:

        
    
          
  • Airflow worker পুরো dedicated K8s pod launch করে training-এর জন্য।
    • 
          
  • Training pod-এ exact GPU resource allocation।
    • 
          
  • Airflow shy task pod-এর status poll।
    • 
          
  • Training fail-এ Airflow logs-এ visible।
    • 
        

        
Alternative — managed compute:

        
    
          
  • SageMakerOperator — SageMaker training job submit, status poll।
    • 
          
  • EMROperator — Spark cluster spin up, training, tear down।
    • 
          
  • Vertex AI training — GCP managed।
    • 
        

        
Async pattern:

        
    
          
  • Airflow task: "submit job", get job ID।
    • 
          
  • Sensor task: poll job status until complete।
    • 
          
  • Worker only briefly used; long wait passive।
    • 
        

        
Checkpointing alignment:

        
    
          
  • Training script checkpoint to S3 every 30 min।
    • 
          
  • Pod fail → new pod, resume from checkpoint।
    • 
          
  • Airflow retry-এ resume natural।
    • 
        

        
BD context — GPU on-prem:

        
    
          
  • K8s cluster on-prem GPU nodes।
    • 
          
  • Airflow workers separate (CPU only)।
    • 
          
  • Training pods K8s scheduled GPU node-এ।
    • 
          
  • Power outage — checkpointing must survive।
    • 
        

        
Cost optimization:

        
    
          
  • Spot/preemptible GPU instance — much cheaper।
    • 
          
  • Aggressive checkpointing essential।
    • 
          
  • "Cost per epoch" tracked।
    • 
        

        

          মূল উপলব্ধি: Long training Airflow worker-এ never run করুন। Pod operator + checkpointing + async sensor — proper pattern। K8s-এ dedicated GPU node — Airflow control-plane only।
        

      

    


    

      
        প্র ০৪
        "Failed pipeline observability — কীভাবে debug দ্রুত?"
      
      

        

          Production pipeline failure-এ minutes matter। Observability infrastructure key।
        

        
Layers of observability:

        
(১) Task logs:

        
    
          
  • Airflow UI per-task log।
    • 
          
  • Stdout/stderr captured।
    • 
          
  • Stored in S3 / file backend।
    • 
          
  • Retain at least 30 days।
    • 
        

        
(২) Structured logging:

        
    
          
  • Tasks log JSON — searchable in ELK / Loki।
    • 
          
  • Tag with run_id, dag_id, execution_date।
    • 
          
  • Error category — easy filter।
    • 
        

        
(৩) Metrics:

        
    
          
  • Task duration over time — regression detect।
    • 
          
  • Failure rate per DAG।
    • 
          
  • Prometheus + Grafana dashboard।
    • 
        

        
(৪) Lineage:

        
    
          
  • "This output came from which task, which run?"।
    • 
          
  • Dagster built-in; Airflow with OpenLineage।
    • 
          
  • Critical when investigating data quality bugs।
    • 
        

        
(৫) Alerting:

        
    
          
  • Failure → Slack/PagerDuty।
    • 
          
  • Granularity: only "important" DAGs alert; others daily digest।
    • 
          
  • Alert fatigue avoid।
    • 
        

        
Common failure scenarios:

        
    
          
  • Upstream data missing → sensor timeout।
    • 
          
  • Schema mismatch → validation task fail।
    • 
          
  • Resource exhaustion (memory) → OOM kill।
    • 
          
  • External API timeout → retry exhausted।
    • 
          
  • Dependency version drift → import error।
    • 
        

        
Debugging speed tips:

        
    
          
  • Reproducible task — local rerun easy।
    • 
          
  • Task-level retry from UI — quick test fix।
    • 
          
  • Log link in alert → 1-click jump।
    • 
          
  • Runbook per common failure।
    • 
        

        
Postmortem culture:

        
    
          
  • Repeated failure → fix root cause, not just retry।
    • 
          
  • "5 whys" analysis।
    • 
          
  • Document and share — team-wide learning।
    • 
        

        

          মূল উপলব্ধি: Pipeline observability — logs + metrics + lineage + alerting layered। Debugging time directly correlates। Invest early in proper alerting + runbooks — incident time slash possible।
        

      

    

  


  

    
 অনুশীলন

    
    
      
  1. 
        Local Airflow: Docker-এ Airflow run করুন। উপরের DAG copy + minimal modify।
        
    
          
          
    
            
    curl -LfO 'https://airflow.apache.org/docs/apache-airflow/stable/docker-compose.yaml' → docker compose up। UI localhost:8080-এ। DAG file dags/ folder-এ drop।
    
          
    
        
    
      
    2. 
      
  2. 
        Idempotent transform: "data/{ds}/processed.csv" pattern। Sample task লিখুন।
        
    
          
          
    
            
    execution_date থেকে directory derive। Output overwrite-safe। Retry-এ same result।
    
          
    
        
    
      
    3. 
      
  3. 
        চিন্তা: Daraz recommendation pipeline — ৬টি step DAG design। Bash command vs Python operator কোথায়?
        
    
          
          
    
            
      
              
    • fetch_data (S3 sync — Bash)।
      • 
              
    • validate (Python schema check)।
      • 
              
    • feature_engineer (Python)।
      • 
              
    • train (KubernetesPodOperator)।
      • 
              
    • evaluate (Python)।
      • 
              
    • register_or_alert (PythonBranchOperator)।
      • 
            
    
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন

    
  







  
  
     এই পাঠে ভুল পেয়েছেন? জানান
  





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ১২ · Feature stores