পাঠ ১৩ · ৩৩-এর মধ্যে · মডিউল ২
Home / AI Courses / MLOps / Training orchestration

Training pipeline orchestration

Training orchestration — Airflow, Prefect, Dagster
৭ মিনিট পড়া উচ্চ · Advanced Python DAG

এই পাঠে যা শিখবেন

  • Workflow orchestrator কেন দরকার — bash/cron-এর সীমা
  • Airflow DAG structure — task, operator, dependency
  • Idempotent design — retry-safe
  • Tool comparison — Airflow vs Prefect vs Dagster vs Argo

১ · কেন orchestrator

একটি ML training pipeline-এ multiple step: data fetch → validate → preprocess → feature engineer → train → evaluate → register → notify. প্রত্যেক-step-এ failure possible. Cron + bash দিয়ে retry, alerting, dependency, lineage — সব manual.

  • Without orchestrator: bash script with bash chains, cron schedule. Failure-এ silent.
  • With orchestrator: DAG, retry, alert, lineage tracking, UI.

২ · DAG fundamentals

DAGDAG (Directed Acyclic Graph)edges directional, no cycle. Workflow এর mathematical model — A → B → C, কোনো backward loop না। = task-গুলোর dependency graph. Cycle না (no infinite loop).

৩ · Airflow DAG example

Python · Airflow DAG 
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.providers.amazon.aws.operators.s3 import S3ListOperator
from datetime import datetime, timedelta

default_args = {
    "owner": "ml-team",
    "retries": 2,
    "retry_delay": timedelta(minutes=5),
    "email_on_failure": True,
}

with DAG(
    dag_id="bangla_sentiment_training",
    schedule="0 2 * * *",   # daily 2 AM
    start_date=datetime(2025, 1, 1),
    default_args=default_args,
    catchup=False,
    tags=["ml", "nlp"],
) as dag:

    def fetch_data(**ctx):
        # data fetch logic
        ds = ctx["ds"]  # execution date
        # download to s3://bucket/data/{ds}/
        return f"s3://bucket/data/{ds}/"

    def validate(**ctx):
        ti = ctx["ti"]
        path = ti.xcom_pull(task_ids="fetch_data")
        # schema, range check
        # raise if fails

    def train(**ctx):
        # train, log to MLflow, register if good
        pass

    def evaluate(**ctx):
        # evaluate against baseline, fairness gate
        pass

    def deploy(**ctx):
        # promote in registry if all gates pass
        pass

    fetch_data_task = PythonOperator(
        task_id="fetch_data", python_callable=fetch_data,
    )
    validate_task = PythonOperator(
        task_id="validate", python_callable=validate,
    )
    train_task = PythonOperator(
        task_id="train", python_callable=train,
    )
    evaluate_task = PythonOperator(
        task_id="evaluate", python_callable=evaluate,
    )
    deploy_task = PythonOperator(
        task_id="deploy", python_callable=deploy,
    )

    fetch_data_task >> validate_task >> train_task >> evaluate_task >> deploy_task

    


    

      Daily 2 AM trigger, retry 2x, failure email. UI-তে DAG visualize, individual task rerun possible.
    


    
৪ · Idempotent task design


    

      Task যে কোনো বার rerun-এ same result দেবে। Otherwise retry harmful.
    


    
    
      
  • File output: overwrite, not append. "data/{ds}/file.csv" — execution-date-keyed.
    • 
      
  • DB write: upsert, not insert.
    • 
      
  • External API: idempotency-key header.
    • 
      
  • Side effects: minimize; explicit cleanup if any.
    • 
    


    

      Anti-pattern: task-এ "append to today's log file". Retry → duplicates. Solution: write to {ds}/output.csv — overwrite.
    


    
৫ · Tool comparison


    
    
      
  • Airflow (২০১৪, Apache):
        
      
          
    • Most mature, widest community.
      • 
          
    • Python-defined DAG, rich UI, scheduler.
      • 
          
    • Used by Airbnb, Pinterest, Uber, BD top-tech.
      • 
          
    • Drawback: heavyweight, dynamic DAG awkward.
      • 
        
    
      
    • 
      
  • Prefect (২০১৮):
        
      
          
    • "Negative engineering" focus — failure-first design.
      • 
          
    • Python-native flow, dynamic mapping easier.
      • 
          
    • Hybrid execution: cloud + local agent.
      • 
          
    • Drawback: smaller community, recent v2 breaking changes.
      • 
        
    
      
    • 
      
  • Dagster (২০১৯):
        
      
          
    • Asset-centric — outputs first-class.
      • 
          
    • Type-checked, lineage built-in.
      • 
          
    • Best for data-heavy pipelines.
      • 
          
    • Drawback: learning curve, less ML-specific.
      • 
        
    
      
    • 
      
  • Argo Workflows (K8s-native):
        
      
          
    • YAML-defined, container-per-step.
      • 
          
    • Best for K8s shop, GPU-heavy training.
      • 
          
    • Drawback: YAML verbose, less Python-friendly.
      • 
        
    
      
    • 
      
  • Kubeflow Pipelines: Lesson 14 — ML-specific Argo.
    • 
    


    
৬ · Scheduling patterns


    
    
      
  • Cron schedule: "0 2 * * *" — daily 2 AM.
    • 
      
  • Event-driven: S3 file landing → trigger.
    • 
      
  • Sensor: wait for upstream — file, DB row, API condition.
    • 
      
  • Manual: UI button — ad-hoc retraining.
    • 
      
  • Backfill: historical date range catch-up.
    • 
    


    
৭ · Long-running training jobs


    
    
      
  • Airflow worker process-এ ৬ ঘণ্টা training — বাজে। Worker tied up.
    • 
      
  • Pattern: KubernetesPodOperator — separate pod-এ training, Airflow শুধু trigger ও wait.
    • 
      
  • Bigger jobs: SparkSubmitOperator, EMR operator — external compute.
    • 
      
  • Status polling — async wait.
    • 
    


    

      
        
          
            
          
        

        ML training DAG — daily schedule
        Idempotent steps with retry + alerts

        
        Fetch
        S3 sync

        
        Validate
        schema, drift

        
        Train
        K8s GPU pod

        
        Evaluate
        vs baseline

        
        Register
        if pass gates

        
        Alert (Slack)
        on failure

        
        MLflow logging
        runs, metrics

        
        
        
        

        
        
      
      
Airflow DAG-এর task chain: success path (left to right), failure → Slack alert, training → MLflow logging side-effect.

    


    

      Bangladesh-এ Airflow most-adopted. managed (Astronomer, MWAA) cost matters — many BD teams self-host on K8s.
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      
        প্র ০১
        "Airflow vs Prefect vs Dagster — Bangladesh team choose কী criteria-তে?"
      
      

        

          Tool choice — team skill, ecosystem fit, ও specific need-এর function.
        

        
Choose Airflow when:

        
    
          
  • Existing Hadoop/Spark/HDFS ecosystem.
    • 
          
  • Team prefers established, well-documented.
    • 
          
  • Many SQL-based pipelines.
    • 
          
  • Operators-এর rich library matters (Snowflake, Redshift, etc.)।
    • 
        

        
Choose Prefect when:

        
    
          
  • Pythonic flow design preferred.
    • 
          
  • Dynamic DAG (loop over variable items) common.
    • 
          
  • Hybrid cloud + local execution.
    • 
          
  • Team comfortable with newer tool.
    • 
        

        
Choose Dagster when:

        
    
          
  • Data lineage critical.
    • 
          
  • Asset-based thinking fits team.
    • 
          
  • Type checking valued.
    • 
          
  • Modern data stack (dbt, Airbyte).
    • 
        

        
Choose Argo when:

        
    
          
  • K8s-native shop.
    • 
          
  • GPU-heavy training, container-per-step preference.
    • 
          
  • Existing K8s expertise.
    • 
        

        
Bangladesh adoption (rough):

        
    
          
  • Airflow: 70%.
    • 
          
  • Prefect: 15%.
    • 
          
  • Dagster: 5%.
    • 
          
  • Argo + KFP: 10%.
    • 
        

        
Migration considerations:

        
    
          
  • Costly — DAG rewrite.
    • 
          
  • "Hybrid" possible — old DAGs Airflow, new Prefect.
    • 
          
  • Don't migrate without clear benefit.
    • 
        

        
Cost:

        
    
          
  • All free OSS.
    • 
          
  • Managed: MWAA $300+/month, Prefect Cloud $/user, Dagster Cloud also $।
    • 
          
  • Self-hosted: Airflow heavyweight (Postgres + scheduler + workers); Prefect simpler.
    • 
        

        

          মূল উপলব্ধি: Default Airflow for most BD teams. Prefect for newer Python-heavy team. Dagster niche-perfect for data engineering-heavy. Argo K8s-shop-এ। Choose carefully — switching cost real.
        

      

    


    

      
        প্র ০২
        "Cron vs event-driven trigger — কখন কোনটা?"
      
      

        

          Trigger choice — predictability vs reactivity trade-off.
        

        
Cron — predictable, scheduled:

        
    
          
  • Daily/hourly retraining.
    • 
          
  • Resource planning easy.
    • 
          
  • Cost predictable.
    • 
          
  • "2 AM run, finish by 4 AM" — operational rhythm.
    • 
        

        
Cron drawbacks:

        
    
          
  • Run-when-no-data — wasted compute.
    • 
          
  • Data ready at 1:30 AM — wait until 2 AM cron.
    • 
        

        
Event-driven — reactive:

        
    
          
  • S3 file landed → trigger immediately.
    • 
          
  • API webhook → trigger.
    • 
          
  • Drift alert → retraining trigger.
    • 
        

        
Event-driven drawbacks:

        
    
          
  • Burst load — many events same time.
    • 
          
  • Debugging harder — when did this trigger?
    • 
          
  • Failure mode — event missed, no run.
    • 
        

        
Hybrid common:

        
    
          
  • Daily cron-based training (default, predictable).
    • 
          
  • Event-driven on top — drift detected → ad-hoc retrain.
    • 
          
  • Sensor pattern — DAG sit waiting until upstream data lands, then proceed.
    • 
        

        
Bangladesh examples:

        
    
          
  • bKash fraud — event-driven (transaction stream).
    • 
          
  • Daraz recommendation — daily cron retraining.
    • 
          
  • Pathao surge — both: daily baseline + event-driven peaks.
    • 
        

        
Idempotency critical for events:

        
    
          
  • Event delivered twice — handle gracefully.
    • 
          
  • Idempotency key in payload.
    • 
          
  • Dedup table.
    • 
        

        

          মূল উপলব্ধি: Cron = train regularly, event = train when needed. Most production systems hybrid. Start cron-only; add event-trigger for specific spike scenarios.
        

      

    


    

      
        প্র ০৩
        "Long training jobs (10+ hours) Airflow-এ — proper pattern কী?"
      
      

        

          Long training Airflow worker-এ run করা = anti-pattern.
        

        
Why anti-pattern:

        
    
          
  • Worker process tied up — other DAGs blocked.
    • 
          
  • Worker restart — training lost.
    • 
          
  • Resource sharing impossible — can't right-size GPU.
    • 
        

        
Right pattern — KubernetesPodOperator:

        
    
          
  • Airflow worker পুরো dedicated K8s pod launch করে training-এর জন্য।
    • 
          
  • Training pod-এ exact GPU resource allocation.
    • 
          
  • Airflow shy task pod-এর status poll.
    • 
          
  • Training fail-এ Airflow logs-এ visible.
    • 
        

        
Alternative — managed compute:

        
    
          
  • SageMakerOperator — SageMaker training job submit, status poll.
    • 
          
  • EMROperator — Spark cluster spin up, training, tear down.
    • 
          
  • Vertex AI training — GCP managed.
    • 
        

        
Async pattern:

        
    
          
  • Airflow task: "submit job", get job ID.
    • 
          
  • Sensor task: poll job status until complete.
    • 
          
  • Worker only briefly used; long wait passive.
    • 
        

        
Checkpointing alignment:

        
    
          
  • Training script checkpoint to S3 every 30 min.
    • 
          
  • Pod fail → new pod, resume from checkpoint.
    • 
          
  • Airflow retry-এ resume natural.
    • 
        

        
BD context — GPU on-prem:

        
    
          
  • K8s cluster on-prem GPU nodes.
    • 
          
  • Airflow workers separate (CPU only).
    • 
          
  • Training pods K8s scheduled GPU node-এ।
    • 
          
  • Power outage — checkpointing must survive.
    • 
        

        
Cost optimization:

        
    
          
  • Spot/preemptible GPU instance — much cheaper.
    • 
          
  • Aggressive checkpointing essential.
    • 
          
  • "Cost per epoch" tracked.
    • 
        

        

          মূল উপলব্ধি: Long training Airflow worker-এ never run করুন। Pod operator + checkpointing + async sensor — proper pattern. K8s-এ dedicated GPU node — Airflow control-plane only.
        

      

    


    

      
        প্র ০৪
        "Failed pipeline observability — কীভাবে debug দ্রুত?"
      
      

        

          Production pipeline failure-এ minutes matter. Observability infrastructure key.
        

        
Layers of observability:

        
(১) Task logs:

        
    
          
  • Airflow UI per-task log.
    • 
          
  • Stdout/stderr captured.
    • 
          
  • Stored in S3 / file backend.
    • 
          
  • Retain at least 30 days.
    • 
        

        
(২) Structured logging:

        
    
          
  • Tasks log JSON — searchable in ELK / Loki.
    • 
          
  • Tag with run_id, dag_id, execution_date.
    • 
          
  • Error category — easy filter.
    • 
        

        
(৩) Metrics:

        
    
          
  • Task duration over time — regression detect.
    • 
          
  • Failure rate per DAG.
    • 
          
  • Prometheus + Grafana dashboard.
    • 
        

        
(৪) Lineage:

        
    
          
  • "This output came from which task, which run?"।
    • 
          
  • Dagster built-in; Airflow with OpenLineage.
    • 
          
  • Critical when investigating data quality bugs.
    • 
        

        
(৫) Alerting:

        
    
          
  • Failure → Slack/PagerDuty.
    • 
          
  • Granularity: only "important" DAGs alert; others daily digest.
    • 
          
  • Alert fatigue avoid.
    • 
        

        
Common failure scenarios:

        
    
          
  • Upstream data missing → sensor timeout.
    • 
          
  • Schema mismatch → validation task fail.
    • 
          
  • Resource exhaustion (memory) → OOM kill.
    • 
          
  • External API timeout → retry exhausted.
    • 
          
  • Dependency version drift → import error.
    • 
        

        
Debugging speed tips:

        
    
          
  • Reproducible task — local rerun easy.
    • 
          
  • Task-level retry from UI — quick test fix.
    • 
          
  • Log link in alert → 1-click jump.
    • 
          
  • Runbook per common failure.
    • 
        

        
Postmortem culture:

        
    
          
  • Repeated failure → fix root cause, not just retry.
    • 
          
  • "5 whys" analysis.
    • 
          
  • Document and share — team-wide learning.
    • 
        

        

          মূল উপলব্ধি: Pipeline observability — logs + metrics + lineage + alerting layered. Debugging time directly correlates. Invest early in proper alerting + runbooks — incident time slash possible.
        

      

    

  


  

    
 অনুশীলন

    
    
      
  1. 
        Local Airflow: Docker-এ Airflow run করুন। উপরের DAG copy + minimal modify.
        
    
          
          
    
            
    curl -LfO 'https://airflow.apache.org/docs/apache-airflow/stable/docker-compose.yaml' → docker compose up. UI localhost:8080-এ। DAG file dags/ folder-এ drop.
    
          
    
        
    
      
    2. 
      
  2. 
        Idempotent transform: "data/{ds}/processed.csv" pattern. Sample task লিখুন।
        
    
          
          
    
            
    execution_date থেকে directory derive. Output overwrite-safe. Retry-এ same result.
    
          
    
        
    
      
    3. 
      
  3. 
        চিন্তা: Daraz recommendation pipeline — ৬টি step DAG design. Bash command vs Python operator কোথায়?
        
    
          
          
    
            
      
              
    • fetch_data (S3 sync — Bash).
      • 
              
    • validate (Python schema check).
      • 
              
    • feature_engineer (Python).
      • 
              
    • train (KubernetesPodOperator).
      • 
              
    • evaluate (Python).
      • 
              
    • register_or_alert (PythonBranchOperator).
      • 
            
    
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন

    
  







  
  
     এই পাঠে ভুল পেয়েছেন? জানান
  





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ১২ · Feature stores