পাঠ ২১ · ৩৩-এর মধ্যে · মডিউল ৩
Home / AI Courses / MLOps / CI/CD for ML

CI/CD for ML — GitHub Actions

CI/CD for ML with GitHub Actions
৮ মিনিট পড়া মধ্য · Intermediate YAML

এই পাঠে যা শিখবেন

  • ML-specific CI test types
  • GitHub Actions workflow syntax
  • Model promotion automation
  • Secrets, caching, cost optimization

১ · ML CI different কেন

Standard software CI: lint → unit test → integration test → build → deploy. ML CI extra dimensions.

  • Data validation: schema, distribution, range.
  • Model performance: baseline-এর সাথে compare.
  • Fairness gate: subgroup metrics.
  • Resource: GPU runner expensive; CPU-only-এর alternative.

২ · Complete GitHub Actions workflow

YAML · .github/workflows/ml.yml 
name: ML pipeline
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
  schedule:
    - cron: '0 2 * * *'  # nightly retrain
  workflow_dispatch:

env:
  PYTHON_VERSION: "3.11"
  AWS_REGION: ap-south-1

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: ${{ env.PYTHON_VERSION }} }
      - uses: actions/cache@v4
        with:
          path: ~/.cache/pip
          key: pip-${{ hashFiles('requirements.lock') }}
      - run: pip install -r requirements.lock
      - run: pytest tests/unit/ -v

  data-validation:
    runs-on: ubuntu-latest
    needs: unit-tests
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: ${{ env.PYTHON_VERSION }} }
      - run: pip install -r requirements.lock
      - name: DVC pull
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: dvc pull data/
      - run: python scripts/validate_data.py data/

  train-and-evaluate:
    runs-on: [self-hosted, gpu]    # ⭐ GPU runner
    needs: data-validation
    if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.lock
      - name: DVC pull
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: dvc pull data/
      - name: Train
        env:
          MLFLOW_TRACKING_URI: ${{ secrets.MLFLOW_URI }}
        run: python src/train.py
      - name: Performance regression test
        run: python scripts/check_performance.py --baseline-auc 0.92
      - name: Fairness gate
        run: python scripts/fairness_check.py --max-disparity 0.05

  build-image:
    runs-on: ubuntu-latest
    needs: train-and-evaluate
    if: success()
    steps:
      - uses: actions/checkout@v4
      - uses: docker/login-action@v3
        with:
          registry: ${{ secrets.REGISTRY_URL }}
          username: ${{ secrets.REGISTRY_USER }}
          password: ${{ secrets.REGISTRY_PASS }}
      - uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: |
            ${{ secrets.REGISTRY_URL }}/sentiment:${{ github.sha }}
            ${{ secrets.REGISTRY_URL }}/sentiment:latest
          cache-from: type=registry,ref=${{ secrets.REGISTRY_URL }}/sentiment:buildcache
          cache-to: type=registry,ref=${{ secrets.REGISTRY_URL }}/sentiment:buildcache,mode=max

  deploy-staging:
    runs-on: ubuntu-latest
    needs: build-image
    environment: staging   # ⭐ environment-protection
    steps:
      - uses: actions/checkout@v4
      - uses: azure/setup-kubectl@v3
      - run: |
          kubectl set image deployment/sentiment-api \
            sentiment=${{ secrets.REGISTRY_URL }}/sentiment:${{ github.sha }} \
            -n staging
          kubectl rollout status deployment/sentiment-api -n staging

    


    

      Pattern features: matrix-aware caching, secrets, GPU self-hosted runner, conditional training (schedule/manual only), environment protection (deploy-staging requires approval).
    


    
৩ · ML-specific tests


    
    
      
  • Unit: preprocess function, custom transform — fast.
    • 
      
  • Data validation: Great Expectations / Pandera — schema, range, missing.
    • 
      
  • Performance regression: "AUC ≥ baseline + tolerance".
    • 
      
  • Fairness: "subgroup accuracy gap < threshold".
    • 
      
  • Schema match: training input == serving input.
    • 
      
  • Smoke test: sample inference end-to-end.
    • 
      
  • Latency benchmark: p99 < SLA.
    • 
    


    
৪ · Secrets management


    
    
      
  • GitHub Actions Secrets — encrypted at rest, exposed as env var.
    • 
      
  • For sensitive: external secret manager (AWS Secrets Manager, Vault) + GH Action fetch.
    • 
      
  • Never echo secret in logs.
    • 
      
  • Rotation periodic.
    • 
    


    
৫ · Self-hosted runners


    
GitHub-hosted runners CPU-only. GPU training-এর জন্য self-hosted.

    
    
      
  • EC2/on-prem GPU machine — runner agent install.
    • 
      
  • Labels: self-hosted, gpu, a100.
    • 
      
  • Workflow-এ runs-on: [self-hosted, gpu].
    • 
      
  • Security: ephemeral runner per job (containers).
    • 
    


    
৬ · Cost optimization


    
    
      
  • Caching: pip, Docker layer, DVC cache.
    • 
      
  • Conditional jobs: path filter — code change only triggers build, doc change skips.
    • 
      
  • Concurrent limit: avoid burst on PR open.
    • 
      
  • Self-hosted: 70-90% cheaper for high-utilization.
    • 
      
  • Spot instances: on-demand spot interrupt-tolerant.
    • 
    


    

      
        
        ML CI/CD pipeline — code commit থেকে production

        
        git push
        PR / main

        
        Unit + lint
        2 min

        
        Data validate
        schema, drift

        
        Train + eval
        GPU runner

        
        Performance gate
        baseline check

        
        Fairness gate

        
        Build image

        
        Deploy staging

        
        Manual approval
        prod

        
        Canary 5%

        
        Full prod

        
        
        
        
        
        
        
        
        
        
      
      
CI pipeline — fail-fast: each gate stop pipeline; production reach hard intentionally.

    


    

      Bangladesh-এ GitHub Actions বহুল-adopted. GitLab CI alternative. On-prem GitLab self-hosted (compliance shop).
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      প্র ০১"PR-এ কী test, nightly-তে কী test?"
      

        
CI cost ও speed-এর জন্য test stratification.

        
Per-PR (fast, cheap):

        
    
          
  • Lint, unit test (Python function).
    • 
          
  • Schema validation.
    • 
          
  • Mock-based integration test.
    • 
          
  • Build container (cache-aware).
    • 
          
  • Total: 5-15 মিনিট।
    • 
        

        
Nightly (expensive):

        
    
          
  • Full training run.
    • 
          
  • Performance regression.
    • 
          
  • Fairness audit.
    • 
          
  • Latency benchmark.
    • 
          
  • Total: 1-3 ঘণ্টা।
    • 
        

        
Pre-merge (manual trigger):

        
    
          
  • Major change PR-এ explicit "/test full" comment.
    • 
          
  • Run nightly-suite.
    • 
          
  • Trade-off: PR slow, but catch issues early.
    • 
        

        
Path-based filter:

        
    
          
  • Doc only — skip CI.
    • 
          
  • Code only — skip data validation.
    • 
          
  • Workflow change — full test.
    • 
        

        
মূল উপলব্ধি: Cost-conscious CI = stratification + caching + filter. Per-PR fast feedback; nightly comprehensive. Big change manual trigger.

      

    


    

      প্র ০২"Flaky model tests — কীভাবে handle?"
      

        
Statistical model — same code, same data, slightly different result possible. Flaky test painful.

        
Sources of flakiness:

        
    
          
  • Random seed not pinned.
    • 
          
  • Hardware non-determinism (GPU).
    • 
          
  • Test data sampling.
    • 
          
  • External dependency (Feature Store, S3).
    • 
        

        
Mitigations:

        
    
          
  • Seed everything (Lesson 5).
    • 
          
  • Tolerance-based assertion: assert abs(auc - 0.92) < 0.005 (not exact).
    • 
          
  • Statistical test: 10 runs avg ± std.
    • 
          
  • Snapshot test fixed dataset.
    • 
        

        
"Quarantine flaky" pattern:

        
    
          
  • Mark flaky test, don't fail PR.
    • 
          
  • Track separately, fix sprint-level.
    • 
          
  • Don't ignore; address as tech debt.
    • 
        

        
Retry policy:

        
    
          
  • 3-retry max, fail if all fail.
    • 
          
  • Random seed-different retries — better statistical confidence.
    • 
        

        
মূল উপলব্ধি: Statistical CI tests — tolerance + reproducibility + quarantine. Pretending no flakes — eventually false alarms erode trust.

      

    


    

      প্র ০৩"GH Actions GPU runner self-host vs cloud — Bangladesh team?"
      

        
GPU CI cost — Bangladesh teams strategic decision.

        
Cloud GPU runner (e.g., GitHub Larger runners):

        
    
          
  • $0.07/min for A100 ≈ $4.20/hour.
    • 
          
  • 2-hour nightly = $8/run × 30 = $240/month.
    • 
          
  • Pros: zero ops, on-demand.
    • 
          
  • Cons: cost compounds, queue at busy times.
    • 
        

        
Self-hosted EC2/on-prem GPU:

        
    
          
  • EC2 g5.xlarge spot: $0.5/hour.
    • 
          
  • 2-hour run × 30 days = $30/month + ops.
    • 
          
  • Pros: cheap, controlled.
    • 
          
  • Cons: maintenance, security.
    • 
        

        
On-prem GPU (existing):

        
    
          
  • Cost: amortized hardware + power + cooling.
    • 
          
  • Bangladesh power outage — UPS, backup.
    • 
          
  • Idle time wasted (CI mostly nighttime).
    • 
        

        
Hybrid pattern:

        
    
          
  • GH-hosted CPU for normal CI.
    • 
          
  • Self-hosted GPU for nightly training.
    • 
          
  • Spot instance for cost optimization.
    • 
        

        
Security:

        
    
          
  • Ephemeral runners — delete after job.
    • 
          
  • Don't run untrusted PR on self-hosted (CI escape risk).
    • 
        

        
মূল উপলব্ধি: High-utilization → self-hosted. Low-utilization → cloud. Bangladesh hybrid common — on-prem dev runner, cloud burst for peaks.

      

    


    

      প্র ০৪"Promotion gates — কত strict, কত flexible?"
      

        
Gates too strict — innovation slow. Too lax — bad model production-এ। Balance critical.

        
Strict gates (must pass):

        
    
          
  • Schema match — training input == serving input.
    • 
          
  • Smoke test — sample inference works.
    • 
          
  • Latency < SLA.
    • 
          
  • Critical fairness (legal compliance).
    • 
        

        
Flexible gates (warning, not block):

        
    
          
  • Slight accuracy drop (within tolerance).
    • 
          
  • Subgroup performance variance (within target).
    • 
          
  • Computational efficiency.
    • 
        

        
Override mechanism:

        
    
          
  • Lead-DS approval comment — bypass for specific PR.
    • 
          
  • Documented in PR description "why bypass".
    • 
          
  • Audit log of overrides.
    • 
        

        
Calibration over time:

        
    
          
  • Initial gates conservative.
    • 
          
  • Loosen as confidence builds.
    • 
          
  • Tighten if incidents.
    • 
        

        
BD context — fintech:

        
    
          
  • Bangladesh Bank compliance — strict gates non-negotiable.
    • 
          
  • Internal-only model — looser.
    • 
        

        
মূল উপলব্ধি: Gate strictness business risk-driven. Start strict, calibrate over time. Override mechanism essential — pure-strict eventually shipped around.

      

    

  


  

    
 অনুশীলন

    
    
      
  1. Setup: উপরের workflow নিজের project-এ adapt. Secrets manage.
        
    Repo settings → Secrets and variables → Actions. AWS_ACCESS_KEY_ID etc. add.
    
      
    2. 
      
  2. Performance gate: python script লিখুন যা MLflow latest run-এর AUC compare.
        
    mlflow.search_runs দিয়ে latest + baseline AUC. Tolerance check. sys.exit(1) on fail.
    
      
    3. 
      
  3. চিন্তা: ৩-environment (dev/stg/prod) workflow design করুন।
        
    
          
      
            
    • dev: every PR auto deploy + smoke test.
      • 
            
    • staging: main merge → auto deploy + integration test.
      • 
            
    • prod: tag push → environment approval (lead-DS) → canary 5% → full.
      • 
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন

    
  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ২০ · Model optimization