পাঠ ৩২ · ৩৩-এর মধ্যে · মডিউল ৪
প্রজেক্ট: end-to-end MLOps system
Capstone — Bangla sentiment, full production stack
এই পাঠে যা শিখবেন
- সব মডিউলের concepts একসাথে integrate
- Real production architecture
- Each tool-এর role + interaction
- Step-by-step implementation walkthrough
১ · Project overview
Goal: Bangla product review (Daraz/Pickaboo style)-এ sentiment classify (positive/negative/neutral)। Production-grade MLOps practices apply।
২ · Architecture
৩ · Step 1 — Data ingestion + DVC
Sources: Daraz public review scrape + Pickaboo + Foodpanda। Manual labeling Bangla annotators (positive/negative/neutral)।
- Repo structure:
data/raw/,data/processed/,models/। - S3 (Mumbai) DVC remote।
- Daily incremental scrape add।
৪ · Step 2 — Training pipeline (Airflow)
- fetch_data → DVC pull।
- validate_data → Great Expectations schema, range, missing।
- preprocess → Bangla tokenize (BanglaBERT tokenizer)।
- train → fine-tune BanglaBERT on annotated data, MLflow autolog।
- evaluate → vs baseline (last week's), fairness across product categories।
- register → if pass gates, MLflow Registry "challenger" alias।
৫ · Step 3 — Dockerfile
Dockerfile
FROM python:3.11.7-slim AS builder
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential gcc && rm -rf /var/lib/apt/lists/*
WORKDIR /build
COPY requirements.lock .
RUN pip wheel --no-cache-dir -r requirements.lock -w /wheels
FROM python:3.11.7-slim AS runtime
RUN apt-get update && apt-get install -y --no-install-recommends \
libgomp1 && rm -rf /var/lib/apt/lists/*
RUN useradd --create-home --shell /bin/bash mluser
WORKDIR /app
COPY --from=builder /wheels /wheels
RUN pip install --no-cache-dir --no-index --find-links=/wheels /wheels/*.whl \
&& rm -rf /wheels
COPY --chown=mluser:mluser app/ /app/
USER mluser
EXPOSE 8000
HEALTHCHECK CMD curl -f http://localhost:8000/healthz || exit 1
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
Multi-stage, non-root, slim, healthcheck — Lesson 6-এর সব pattern।
৬ · Step 4 — K8s deployment
YAML · K8s
apiVersion: apps/v1
kind: Deployment
metadata:
name: bangla-sentiment
labels: { app: bangla-sentiment, model-version: v1.3.0 }
spec:
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate: { maxUnavailable: 0, maxSurge: 1 }
selector:
matchLabels: { app: bangla-sentiment }
template:
metadata:
labels: { app: bangla-sentiment, model-version: v1.3.0 }
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8000"
spec:
containers:
- name: api
image: registry.example.com/bangla-sentiment:v1.3.0
ports: [{ containerPort: 8000 }]
env:
- { name: MODEL_VERSION, value: "v1.3.0" }
- { name: MLFLOW_TRACKING_URI, value: "http://mlflow:5000" }
resources:
requests: { cpu: "1", memory: "2Gi" }
limits: { cpu: "2", memory: "4Gi" }
readinessProbe:
httpGet: { path: /readyz, port: 8000 }
initialDelaySeconds: 30
periodSeconds: 5
livenessProbe:
httpGet: { path: /healthz, port: 8000 }
initialDelaySeconds: 60
periodSeconds: 30
---
apiVersion: v1
kind: Service
metadata: { name: bangla-sentiment }
spec:
selector: { app: bangla-sentiment }
ports: [{ port: 80, targetPort: 8000 }]
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: bangla-sentiment }
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: bangla-sentiment
minReplicas: 3
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target: { type: Utilization, averageUtilization: 70 }
Deployment + Service + HPA। Resource limit, probe, version label, Prometheus annotation। Lessons 7, 16, 27 patterns combined।
৭ · Step 5 — GitHub Actions CI/CD
(Lesson 21 reference) — pipeline:
- PR-এ: lint + unit test + smoke build।
- main merge-এ: train + evaluate + build image + push registry + deploy staging।
- Manual approval (lead-DS) → production canary।
- Argo Rollouts canary 5%→25%→100% (Lesson 22)।
৮ · Step 6 — Monitoring + alerts
- Prometheus scrape FastAPI
/metrics+ custom drift PSI metric। - Grafana dashboards: latency, error, prediction distribution, per-feature PSI।
- Alerts: error > 1%, p99 > 200ms, PSI > 0.25।
- Alertmanager → Slack #ml-alerts।
৯ · Step 7 — Retraining loop
- Drift detected → Airflow trigger immediately।
- Otherwise: weekly scheduled retrain (Sunday 2 AM)।
- Champion-challenger: new model "challenger" alias।
- Shadow deploy 24 hours।
- If challenger metric better → canary।
- If canary OK → promote champion।
১০ · Step 8 — Cost overview (BD)
- K8s cluster (3 nodes, GKE Mumbai): ~$300/month।
- S3 (Mumbai) data + artifact: ~$50/month।
- MLflow (small Postgres + MinIO): ~$50/month।
- GH Actions (free tier mostly): $0।
- GPU training nightly (spot): ~$50/month।
- Total: ~$450/month = ~৫০,০০০ BDT/month।
১১ · Step 9 — Common production issues
- Model drift (Bangla slang evolution) — quarterly retrain catches।
- Mumbai latency BD users — ~50ms RTT acceptable।
- Eid traffic spike — HPA scale up automatic।
- Bangladesh power outage on-prem fallback — cloud primary।
১২ · Step 10 — Team handoff
- Documentation: README, runbooks, on-call rotation।
- "How to retrain" doc — clear steps।
- Incident postmortem template।
- Quarterly architecture review meeting।
এই project complete করলে — production MLOps engineer হিসেবে portfolio-তে শক্ত addition। Bangladesh job market-এ বিশেষ valuable।
ভাবনার প্রশ্ন
প্র ০১"Project scope creep — কীভাবে control?"
Capstone-এ scope explosion common।
"Just one more feature":
- Add A/B test framework।
- Add multi-model ensemble।
- Add explainability।
- Add advanced security।
Each individually valuable; together — never finishing।
Strategies:
- "MVP first" — simplest end-to-end work।
- Iterate — week 1 baseline, week 2 polish, etc.।
- Time-box each phase।
- "Out-of-scope" backlog — deliberate decision।
BD pragmatic:
- Production-quality MVP > 80% complete masterpiece।
- "Shipped" portfolio item beats "in-progress for 6 months"।
মূল উপলব্ধি: Scope discipline = career skill। Time-box, ship, iterate। Perfectionism enemy of done।
প্র ০২"Team handoff — solo project থেকে team production-এ?"
Solo MVP team-হাতে handover challenges।
Required artifacts:
- README — setup, run, deploy steps।
- Architecture diagram।
- Runbook — common incidents response।
- API documentation।
- "Day-1 onboarding" guide।
Code quality:
- Type hints।
- Docstrings।
- Modular structure।
- Test coverage।
Operational handoff:
- On-call rotation।
- Slack channel।
- Incident escalation chain।
- Quarterly review।
Knowledge transfer:
- Recorded walkthrough video।
- Pair programming session।
- "Why we chose X" decision log।
BD context:
- BD startup — solo founder ML common।
- Successful handoff = startup graduation।
মূল উপলব্ধি: Handoff documentation up-front cost; later dividend। Code self-documenting + structured artifacts। "Bus factor" 1 → 3+।
প্র ০৩"Cost-aware design — early decisions long-term impact?"
Design decisions cost compound।
Cheap-now, expensive-later:
- "Always-on big GPU" — always burning cost।
- "Verbose logging without retention" — TB explosion।
- "Premium tier API for all" — small task overpaid।
- "Multi-region without need" — bandwidth cost।
Cost-aware patterns:
- Spot instance for training।
- Auto-scale down during low traffic।
- Tiered storage (hot/warm/cold)।
- Model routing (cheap → expensive)।
- Cache aggressively।
Monitoring cost:
- Monthly cost dashboard।
- Per-team chargeback।
- Cost spike alert।
BD startup reality:
- Bootstrapped startup — runway awareness।
- $500/month vs $5000/month — 10x runway।
- Don't engineer expensively early।
মূল উপলব্ধি: Cost ≠ afterthought। Architectural decision। Monthly review habit। "Cost-conscious" engineer differentiator BD market-এ।
প্র ০৪"Production debugging — কী tools দরকার?"
Production incident response — speed critical।
Debugging tools:
- Logs — structured (JSON), centralized (ELK, Loki)।
- Traces — distributed (Jaeger, Tempo)।
- Metrics — Prometheus + Grafana।
- Profiler — py-spy, pprof।
kubectl essentials:
- describe pod / events।
- logs --previous (crashed)।
- exec interactive debug।
- top — resource utilization।
Incident workflow:
- Alert → on-call → runbook follow।
- Reproduce locally if possible।
- Quick fix vs root-cause — differentiate।
- Postmortem within 48 hours।
"Hidden tools":
- Curl directly to pod (skip ingress)।
- Port-forward to internal service।
- Trace request through service mesh।
BD context — power outage:
- On-prem partial outage common।
- Multi-region failover essential।
- Drill regularly।
মূল উপলব্ধি: Debug skill = production engineer's edge। Tool fluency + incident discipline। Postmortem culture continuous improve।
অনুশীলন
- Build phase 1: Bangla sentiment train → MLflow log → Docker → local FastAPI serve।
Hugging Face dataset Bangla sentiment। Fine-tune BanglaBERT। MLflow autolog। Dockerfile build, docker run।
- Build phase 2: Add monitoring — Prometheus instrumentation।
prometheus-fastapi-instrumentator + custom prediction metrics। Local Prometheus + Grafana docker compose।
- Build phase 3: Deploy K8s (minikube)। CI workflow GH Actions write।
K8s manifest apply, GH Actions yaml lint + build + push। Manual deploy step initially।