পাঠ ২১ · ৩৩-এর মধ্যে · মডিউল ৩
Home / AI Courses / MLOps / CI/CD for ML

CI/CD for ML — GitHub Actions

CI/CD for ML with GitHub Actions
৮ মিনিট পড়া মধ্য · Intermediate YAML

এই পাঠে যা শিখবেন

  • ML-specific CI test types
  • GitHub Actions workflow syntax
  • Model promotion automation
  • Secrets, caching, cost optimization

১ · ML CI different কেন

Standard software CI: lint → unit test → integration test → build → deploy। ML CI extra dimensions।

  • Data validation: schema, distribution, range।
  • Model performance: baseline-এর সাথে compare।
  • Fairness gate: subgroup metrics।
  • Resource: GPU runner expensive; CPU-only-এর alternative।

২ · Complete GitHub Actions workflow

YAML · .github/workflows/ml.yml 
name: ML pipeline
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
  schedule:
    - cron: '0 2 * * *'  # nightly retrain
  workflow_dispatch:

env:
  PYTHON_VERSION: "3.11"
  AWS_REGION: ap-south-1

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: ${{ env.PYTHON_VERSION }} }
      - uses: actions/cache@v4
        with:
          path: ~/.cache/pip
          key: pip-${{ hashFiles('requirements.lock') }}
      - run: pip install -r requirements.lock
      - run: pytest tests/unit/ -v

  data-validation:
    runs-on: ubuntu-latest
    needs: unit-tests
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: ${{ env.PYTHON_VERSION }} }
      - run: pip install -r requirements.lock
      - name: DVC pull
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: dvc pull data/
      - run: python scripts/validate_data.py data/

  train-and-evaluate:
    runs-on: [self-hosted, gpu]    # ⭐ GPU runner
    needs: data-validation
    if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.lock
      - name: DVC pull
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: dvc pull data/
      - name: Train
        env:
          MLFLOW_TRACKING_URI: ${{ secrets.MLFLOW_URI }}
        run: python src/train.py
      - name: Performance regression test
        run: python scripts/check_performance.py --baseline-auc 0.92
      - name: Fairness gate
        run: python scripts/fairness_check.py --max-disparity 0.05

  build-image:
    runs-on: ubuntu-latest
    needs: train-and-evaluate
    if: success()
    steps:
      - uses: actions/checkout@v4
      - uses: docker/login-action@v3
        with:
          registry: ${{ secrets.REGISTRY_URL }}
          username: ${{ secrets.REGISTRY_USER }}
          password: ${{ secrets.REGISTRY_PASS }}
      - uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: |
            ${{ secrets.REGISTRY_URL }}/sentiment:${{ github.sha }}
            ${{ secrets.REGISTRY_URL }}/sentiment:latest
          cache-from: type=registry,ref=${{ secrets.REGISTRY_URL }}/sentiment:buildcache
          cache-to: type=registry,ref=${{ secrets.REGISTRY_URL }}/sentiment:buildcache,mode=max

  deploy-staging:
    runs-on: ubuntu-latest
    needs: build-image
    environment: staging   # ⭐ environment-protection
    steps:
      - uses: actions/checkout@v4
      - uses: azure/setup-kubectl@v3
      - run: |
          kubectl set image deployment/sentiment-api \
            sentiment=${{ secrets.REGISTRY_URL }}/sentiment:${{ github.sha }} \
            -n staging
          kubectl rollout status deployment/sentiment-api -n staging

    


    

      Pattern features: matrix-aware caching, secrets, GPU self-hosted runner, conditional training (schedule/manual only), environment protection (deploy-staging requires approval)।
    


    
৩ · ML-specific tests


    
    
      
  • Unit: preprocess function, custom transform — fast।
    • 
      
  • Data validation: Great Expectations / Pandera — schema, range, missing।
    • 
      
  • Performance regression: "AUC ≥ baseline + tolerance"।
    • 
      
  • Fairness: "subgroup accuracy gap < threshold"।
    • 
      
  • Schema match: training input == serving input।
    • 
      
  • Smoke test: sample inference end-to-end।
    • 
      
  • Latency benchmark: p99 < SLA।
    • 
    


    
৪ · Secrets management


    
    
      
  • GitHub Actions Secrets — encrypted at rest, exposed as env var।
    • 
      
  • For sensitive: external secret manager (AWS Secrets Manager, Vault) + GH Action fetch।
    • 
      
  • Never echo secret in logs।
    • 
      
  • Rotation periodic।
    • 
    


    
৫ · Self-hosted runners


    
GitHub-hosted runners CPU-only। GPU training-এর জন্য self-hosted।

    
    
      
  • EC2/on-prem GPU machine — runner agent install।
    • 
      
  • Labels: self-hosted, gpu, a100।
    • 
      
  • Workflow-এ runs-on: [self-hosted, gpu]।
    • 
      
  • Security: ephemeral runner per job (containers)।
    • 
    


    
৬ · Cost optimization


    
    
      
  • Caching: pip, Docker layer, DVC cache।
    • 
      
  • Conditional jobs: path filter — code change only triggers build, doc change skips।
    • 
      
  • Concurrent limit: avoid burst on PR open।
    • 
      
  • Self-hosted: 70-90% cheaper for high-utilization।
    • 
      
  • Spot instances: on-demand spot interrupt-tolerant।
    • 
    


    

      
        
        ML CI/CD pipeline — code commit থেকে production

        
        git push
        PR / main

        
        Unit + lint
        2 min

        
        Data validate
        schema, drift

        
        Train + eval
        GPU runner

        
        Performance gate
        baseline check

        
        Fairness gate

        
        Build image

        
        Deploy staging

        
        Manual approval
        prod

        
        Canary 5%

        
        Full prod

        
        
        
        
        
        
        
        
        
        
      
      
CI pipeline — fail-fast: each gate stop pipeline; production reach hard intentionally।

    


    

      Bangladesh-এ GitHub Actions বহুল-adopted। GitLab CI alternative। On-prem GitLab self-hosted (compliance shop)।
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      প্র ০১"PR-এ কী test, nightly-তে কী test?"
      

        
CI cost ও speed-এর জন্য test stratification।

        
Per-PR (fast, cheap):

        
    
          
  • Lint, unit test (Python function)।
    • 
          
  • Schema validation।
    • 
          
  • Mock-based integration test।
    • 
          
  • Build container (cache-aware)।
    • 
          
  • Total: 5-15 মিনিট।
    • 
        

        
Nightly (expensive):

        
    
          
  • Full training run।
    • 
          
  • Performance regression।
    • 
          
  • Fairness audit।
    • 
          
  • Latency benchmark।
    • 
          
  • Total: 1-3 ঘণ্টা।
    • 
        

        
Pre-merge (manual trigger):

        
    
          
  • Major change PR-এ explicit "/test full" comment।
    • 
          
  • Run nightly-suite।
    • 
          
  • Trade-off: PR slow, but catch issues early।
    • 
        

        
Path-based filter:

        
    
          
  • Doc only — skip CI।
    • 
          
  • Code only — skip data validation।
    • 
          
  • Workflow change — full test।
    • 
        

        
মূল উপলব্ধি: Cost-conscious CI = stratification + caching + filter। Per-PR fast feedback; nightly comprehensive। Big change manual trigger।

      

    


    

      প্র ০২"Flaky model tests — কীভাবে handle?"
      

        
Statistical model — same code, same data, slightly different result possible। Flaky test painful।

        
Sources of flakiness:

        
    
          
  • Random seed not pinned।
    • 
          
  • Hardware non-determinism (GPU)।
    • 
          
  • Test data sampling।
    • 
          
  • External dependency (Feature Store, S3)।
    • 
        

        
Mitigations:

        
    
          
  • Seed everything (Lesson 5)।
    • 
          
  • Tolerance-based assertion: assert abs(auc - 0.92) < 0.005 (not exact)।
    • 
          
  • Statistical test: 10 runs avg ± std।
    • 
          
  • Snapshot test fixed dataset।
    • 
        

        
"Quarantine flaky" pattern:

        
    
          
  • Mark flaky test, don't fail PR।
    • 
          
  • Track separately, fix sprint-level।
    • 
          
  • Don't ignore; address as tech debt।
    • 
        

        
Retry policy:

        
    
          
  • 3-retry max, fail if all fail।
    • 
          
  • Random seed-different retries — better statistical confidence।
    • 
        

        
মূল উপলব্ধি: Statistical CI tests — tolerance + reproducibility + quarantine। Pretending no flakes — eventually false alarms erode trust।

      

    


    

      প্র ০৩"GH Actions GPU runner self-host vs cloud — Bangladesh team?"
      

        
GPU CI cost — Bangladesh teams strategic decision।

        
Cloud GPU runner (e.g., GitHub Larger runners):

        
    
          
  • $0.07/min for A100 ≈ $4.20/hour।
    • 
          
  • 2-hour nightly = $8/run × 30 = $240/month।
    • 
          
  • Pros: zero ops, on-demand।
    • 
          
  • Cons: cost compounds, queue at busy times।
    • 
        

        
Self-hosted EC2/on-prem GPU:

        
    
          
  • EC2 g5.xlarge spot: $0.5/hour।
    • 
          
  • 2-hour run × 30 days = $30/month + ops।
    • 
          
  • Pros: cheap, controlled।
    • 
          
  • Cons: maintenance, security।
    • 
        

        
On-prem GPU (existing):

        
    
          
  • Cost: amortized hardware + power + cooling।
    • 
          
  • Bangladesh power outage — UPS, backup।
    • 
          
  • Idle time wasted (CI mostly nighttime)।
    • 
        

        
Hybrid pattern:

        
    
          
  • GH-hosted CPU for normal CI।
    • 
          
  • Self-hosted GPU for nightly training।
    • 
          
  • Spot instance for cost optimization।
    • 
        

        
Security:

        
    
          
  • Ephemeral runners — delete after job।
    • 
          
  • Don't run untrusted PR on self-hosted (CI escape risk)।
    • 
        

        
মূল উপলব্ধি: High-utilization → self-hosted। Low-utilization → cloud। Bangladesh hybrid common — on-prem dev runner, cloud burst for peaks।

      

    


    

      প্র ০৪"Promotion gates — কত strict, কত flexible?"
      

        
Gates too strict — innovation slow। Too lax — bad model production-এ। Balance critical।

        
Strict gates (must pass):

        
    
          
  • Schema match — training input == serving input।
    • 
          
  • Smoke test — sample inference works।
    • 
          
  • Latency < SLA।
    • 
          
  • Critical fairness (legal compliance)।
    • 
        

        
Flexible gates (warning, not block):

        
    
          
  • Slight accuracy drop (within tolerance)।
    • 
          
  • Subgroup performance variance (within target)।
    • 
          
  • Computational efficiency।
    • 
        

        
Override mechanism:

        
    
          
  • Lead-DS approval comment — bypass for specific PR।
    • 
          
  • Documented in PR description "why bypass"।
    • 
          
  • Audit log of overrides।
    • 
        

        
Calibration over time:

        
    
          
  • Initial gates conservative।
    • 
          
  • Loosen as confidence builds।
    • 
          
  • Tighten if incidents।
    • 
        

        
BD context — fintech:

        
    
          
  • Bangladesh Bank compliance — strict gates non-negotiable।
    • 
          
  • Internal-only model — looser।
    • 
        

        
মূল উপলব্ধি: Gate strictness business risk-driven। Start strict, calibrate over time। Override mechanism essential — pure-strict eventually shipped around।

      

    

  


  

    
 অনুশীলন

    
    
      
  1. Setup: উপরের workflow নিজের project-এ adapt। Secrets manage।
        
    Repo settings → Secrets and variables → Actions। AWS_ACCESS_KEY_ID etc. add।
    
      
    2. 
      
  2. Performance gate: python script লিখুন যা MLflow latest run-এর AUC compare।
        
    mlflow.search_runs দিয়ে latest + baseline AUC। Tolerance check। sys.exit(1) on fail।
    
      
    3. 
      
  3. চিন্তা: ৩-environment (dev/stg/prod) workflow design করুন।
        
    
          
      
            
    • dev: every PR auto deploy + smoke test।
      • 
            
    • staging: main merge → auto deploy + integration test।
      • 
            
    • prod: tag push → environment approval (lead-DS) → canary 5% → full।
      • 
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন

    
  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ২০ · Model optimization