পাঠ ৩২ · ৩৩-এর মধ্যে · মডিউল ৪

Home / AI Courses / MLOps / End-to-end project

প্রজেক্ট: end-to-end MLOps system

Capstone — Bangla sentiment, full production stack

১৫ মিনিট পড়া উচ্চ · Advanced Capstone

এই পাঠে যা শিখবেন

সব মডিউলের concepts একসাথে integrate
Real production architecture
Each tool-এর role + interaction
Step-by-step implementation walkthrough

১ · Project overview

Goal: Bangla product review (Daraz/Pickaboo style)-এ sentiment classify (positive/negative/neutral). Production-grade MLOps practices apply.

২ · Architecture

End-to-end production architecture — সব 31 lesson-এর concept integrated.

৩ · Step 1 — Data ingestion + DVC

Sources: Daraz public review scrape + Pickaboo + Foodpanda. Manual labeling Bangla annotators (positive/negative/neutral).

Repo structure: data/raw/, data/processed/, models/।
S3 (Mumbai) DVC remote.
Daily incremental scrape add.

৪ · Step 2 — Training pipeline (Airflow)

fetch_data → DVC pull.
validate_data → Great Expectations schema, range, missing.
preprocess → Bangla tokenize (BanglaBERT tokenizer).
train → fine-tune BanglaBERT on annotated data, MLflow autolog.
evaluate → vs baseline (last week's), fairness across product categories.
register → if pass gates, MLflow Registry "challenger" alias.

৫ · Step 3 — Dockerfile

Dockerfile

FROM python:3.11.7-slim AS builder
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential gcc && rm -rf /var/lib/apt/lists/*
WORKDIR /build
COPY requirements.lock .
RUN pip wheel --no-cache-dir -r requirements.lock -w /wheels

FROM python:3.11.7-slim AS runtime
RUN apt-get update && apt-get install -y --no-install-recommends \
    libgomp1 && rm -rf /var/lib/apt/lists/*
RUN useradd --create-home --shell /bin/bash mluser
WORKDIR /app
COPY --from=builder /wheels /wheels
RUN pip install --no-cache-dir --no-index --find-links=/wheels /wheels/*.whl \
    && rm -rf /wheels
COPY --chown=mluser:mluser app/ /app/
USER mluser
EXPOSE 8000
HEALTHCHECK CMD curl -f http://localhost:8000/healthz || exit 1
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Multi-stage, non-root, slim, healthcheck — Lesson 6-এর সব pattern.

৬ · Step 4 — K8s deployment

YAML · K8s

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bangla-sentiment
  labels: { app: bangla-sentiment, model-version: v1.3.0 }
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate: { maxUnavailable: 0, maxSurge: 1 }
  selector:
    matchLabels: { app: bangla-sentiment }
  template:
    metadata:
      labels: { app: bangla-sentiment, model-version: v1.3.0 }
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8000"
    spec:
      containers:
        - name: api
          image: registry.example.com/bangla-sentiment:v1.3.0
          ports: [{ containerPort: 8000 }]
          env:
            - { name: MODEL_VERSION, value: "v1.3.0" }
            - { name: MLFLOW_TRACKING_URI, value: "http://mlflow:5000" }
          resources:
            requests: { cpu: "1", memory: "2Gi" }
            limits: { cpu: "2", memory: "4Gi" }
          readinessProbe:
            httpGet: { path: /readyz, port: 8000 }
            initialDelaySeconds: 30
            periodSeconds: 5
          livenessProbe:
            httpGet: { path: /healthz, port: 8000 }
            initialDelaySeconds: 60
            periodSeconds: 30
---
apiVersion: v1
kind: Service
metadata: { name: bangla-sentiment }
spec:
  selector: { app: bangla-sentiment }
  ports: [{ port: 80, targetPort: 8000 }]
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: bangla-sentiment }
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: bangla-sentiment
  minReplicas: 3
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: { type: Utilization, averageUtilization: 70 }

Deployment + Service + HPA. Resource limit, probe, version label, Prometheus annotation. Lessons 7, 16, 27 patterns combined.

৭ · Step 5 — GitHub Actions CI/CD

(Lesson 21 reference) — pipeline:

PR-এ: lint + unit test + smoke build.
main merge-এ: train + evaluate + build image + push registry + deploy staging.
Manual approval (lead-DS) → production canary.
Argo Rollouts canary 5%→25%→100% (Lesson 22).

৮ · Step 6 — Monitoring + alerts

Prometheus scrape FastAPI /metrics + custom drift PSI metric.
Grafana dashboards: latency, error, prediction distribution, per-feature PSI.
Alerts: error > 1%, p99 > 200ms, PSI > 0.25.
Alertmanager → Slack #ml-alerts.

৯ · Step 7 — Retraining loop

Drift detected → Airflow trigger immediately.
Otherwise: weekly scheduled retrain (Sunday 2 AM).
Champion-challenger: new model "challenger" alias.
Shadow deploy 24 hours.
If challenger metric better → canary.
If canary OK → promote champion.

১০ · Step 8 — Cost overview (BD)

K8s cluster (3 nodes, GKE Mumbai): ~$300/month.
S3 (Mumbai) data + artifact: ~$50/month.
MLflow (small Postgres + MinIO): ~$50/month.
GH Actions (free tier mostly): $0.
GPU training nightly (spot): ~$50/month.
Total: ~$450/month = ~৫০,০০০ BDT/month.

১১ · Step 9 — Common production issues

Model drift (Bangla slang evolution) — quarterly retrain catches.
Mumbai latency BD users — ~50ms RTT acceptable.
Eid traffic spike — HPA scale up automatic.
Bangladesh power outage on-prem fallback — cloud primary.

১২ · Step 10 — Team handoff

Documentation: README, runbooks, on-call rotation.
"How to retrain" doc — clear steps.
Incident postmortem template.
Quarterly architecture review meeting.

এই project complete করলে — production MLOps engineer হিসেবে portfolio-তে শক্ত addition. Bangladesh job market-এ বিশেষ valuable.

ভাবনার প্রশ্ন

প্র ০১"Project scope creep — কীভাবে control?"

Capstone-এ scope explosion common.

"Just one more feature":

Add A/B test framework.
Add multi-model ensemble.
Add explainability.
Add advanced security.

Each individually valuable; together — never finishing.

Strategies:

"MVP first" — simplest end-to-end work.
Iterate — week 1 baseline, week 2 polish, etc.।
Time-box each phase.
"Out-of-scope" backlog — deliberate decision.

BD pragmatic:

Production-quality MVP > 80% complete masterpiece.
"Shipped" portfolio item beats "in-progress for 6 months".

মূল উপলব্ধি: Scope discipline = career skill. Time-box, ship, iterate. Perfectionism enemy of done.

প্র ০২"Team handoff — solo project থেকে team production-এ?"

Solo MVP team-হাতে handover challenges.

Required artifacts:

README — setup, run, deploy steps.
Architecture diagram.
Runbook — common incidents response.
API documentation.
"Day-1 onboarding" guide.

Code quality:

Type hints.
Docstrings.
Modular structure.
Test coverage.

Operational handoff:

On-call rotation.
Slack channel.
Incident escalation chain.
Quarterly review.

Knowledge transfer:

Recorded walkthrough video.
Pair programming session.
"Why we chose X" decision log.

BD context:

BD startup — solo founder ML common.
Successful handoff = startup graduation.

মূল উপলব্ধি: Handoff documentation up-front cost; later dividend. Code self-documenting + structured artifacts. "Bus factor" 1 → 3+.

প্র ০৩"Cost-aware design — early decisions long-term impact?"

Design decisions cost compound.

Cheap-now, expensive-later:

"Always-on big GPU" — always burning cost.
"Verbose logging without retention" — TB explosion.
"Premium tier API for all" — small task overpaid.
"Multi-region without need" — bandwidth cost.

Cost-aware patterns:

Spot instance for training.
Auto-scale down during low traffic.
Tiered storage (hot/warm/cold).
Model routing (cheap → expensive).
Cache aggressively.

Monitoring cost:

Monthly cost dashboard.
Per-team chargeback.
Cost spike alert.

BD startup reality:

Bootstrapped startup — runway awareness.
$500/month vs $5000/month — 10x runway.
Don't engineer expensively early.

মূল উপলব্ধি: Cost ≠ afterthought. Architectural decision. Monthly review habit. "Cost-conscious" engineer differentiator BD market-এ।

প্র ০৪"Production debugging — কী tools দরকার?"

Production incident response — speed critical.

Debugging tools:

Logs — structured (JSON), centralized (ELK, Loki).
Traces — distributed (Jaeger, Tempo).
Metrics — Prometheus + Grafana.
Profiler — py-spy, pprof.

kubectl essentials:

describe pod / events.
logs --previous (crashed).
exec interactive debug.
top — resource utilization.

Incident workflow:

Alert → on-call → runbook follow.
Reproduce locally if possible.
Quick fix vs root-cause — differentiate.
Postmortem within 48 hours.

"Hidden tools":

Curl directly to pod (skip ingress).
Port-forward to internal service.
Trace request through service mesh.

BD context — power outage:

On-prem partial outage common.
Multi-region failover essential.
Drill regularly.

মূল উপলব্ধি: Debug skill = production engineer's edge. Tool fluency + incident discipline. Postmortem culture continuous improve.

অনুশীলন

Build phase 1: Bangla sentiment train → MLflow log → Docker → local FastAPI serve.
Hugging Face dataset Bangla sentiment. Fine-tune BanglaBERT. MLflow autolog. Dockerfile build, docker run.
Build phase 2: Add monitoring — Prometheus instrumentation.
prometheus-fastapi-instrumentator + custom prediction metrics. Local Prometheus + Grafana docker compose.
Build phase 3: Deploy K8s (minikube). CI workflow GH Actions write.
K8s manifest apply, GH Actions yaml lint + build + push. Manual deploy step initially.