Canary ও blue-green deployment
এই পাঠে যা শিখবেন
- Canary deployment workflow + monitoring
- Blue-green vs canary trade-offs
- Shadow deployment for ML-specific validation
- Auto-rollback triggers
১ · কেন canary/blue-green
"Replace all pods at once" deployment risky — bad model serve all users immediately।
- Canary: gradual exposure — issue detected early, blast radius small।
- Blue-green: instant rollback — new env fail, switch back instantly।
- Shadow: zero user-impact validation।
২ · Canary deployment
Pattern:
- Deploy v2 alongside v1 (small replica count)।
- Route 5% traffic → v2।
- Monitor 30 min: latency, error rate, accuracy।
- If healthy: increase to 25%, 50%, 100%।
- If unhealthy: rollback (route 0% to v2)।
৩ · Argo Rollouts canary
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: sentiment-api
spec:
replicas: 10
strategy:
canary:
steps:
- setWeight: 5 # 5% traffic to canary
- pause: { duration: 30m }
- analysis:
templates:
- templateName: success-rate
- templateName: latency-p99
- setWeight: 25
- pause: { duration: 1h }
- analysis: { templates: [{templateName: success-rate}] }
- setWeight: 50
- pause: { duration: 2h }
- setWeight: 100
analysis:
templates:
- templateName: success-rate
startingStep: 1
args: []
trafficRouting:
istio:
virtualService: { name: sentiment-vs }
selector:
matchLabels: { app: sentiment-api }
template:
metadata:
labels: { app: sentiment-api }
spec:
containers:
- name: api
image: registry/sentiment:v2.0
# ... resources, probes
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: { name: success-rate }
spec:
metrics:
- name: success-rate
interval: 5m
successCondition: result[0] >= 0.99
failureLimit: 2
provider:
prometheus:
address: http://prometheus.monitoring:9090
query: |
sum(rate(http_requests_total{job="sentiment-api",status=~"2.."}[5m]))
/
sum(rate(http_requests_total{job="sentiment-api"}[5m]))৪ · Blue-green
Two complete environments, switch traffic atomically।
- Blue: current production, all traffic।
- Green: new version deploy + warm up + smoke test।
- Switch: ingress flip blue → green। Instant।
- Rollback: flip back। Instant।
Pros: instant switch, instant rollback। Cons: 2× resource cost during deployment।
৫ · Shadow deployment (ML-specific)
New model receives copy of production requests; response logged, not returned। Silent compare।
- Production unaffected — zero risk।
- Real production data — best validation।
- Distribution comparison — old vs new prediction।
- Performance compare without committing।
৬ · Auto-rollback triggers
- Error rate spike: > 1% threshold।
- Latency degradation: p99 > SLA।
- Accuracy drop: ground truth available — accuracy gate।
- Drift signal: input/output distribution shift।
- Business KPI: conversion drop, complaint spike।
৭ · Cost of running both
Canary 5% — 1 small extra pod। Blue-green — full duplicate (during deployment). Plan capacity।
ভাবনার প্রশ্ন
প্র ০১"Canary % split — কীভাবে decide?"
Initial weight + step progression — strategy।
Initial weight (5%, 10%, 1%):
- Critical service (payment, fraud): 1% — minimum exposure।
- Standard (recommendation): 5%।
- Experimental (new feature): 10%।
Step progression:
- 5% → 25% → 50% → 100%: aggressive, fast feedback।
- 5% → 10% → 25% → 50% → 100%: cautious, more confidence।
- 1% → 10% → 100%: bimodal — small or full।
Time per step:
- 30 min — fast metric-driven (latency, error)।
- 1-2 hours — moderate signal need।
- 24 hours — slow signals (drift, conversion rate)।
Statistical sample size:
- 5% × 30 min × 1000 RPS = 90,000 requests — strong signal।
- 1% × 30 min × 100 RPS = 1,800 requests — weak। Increase % or duration।
BD fintech example:
- bKash fraud canary: 1% → 5% → 25% → 100%, each step 2 hours, total 8 hours।
- Conservative due to financial impact।
মূল উপলব্ধি: Canary % depends risk profile + signal volume + business tolerance। Critical service = small initial + slow steps; experimental = aggressive।
প্র ০২"Rollback — কোন metric trigger?"
Rollback decision metric-driven, fast।
Hard triggers (immediate rollback):
- Error rate > 1% (vs < 0.1% baseline)।
- p99 latency > 2× baseline।
- Pod crash loop।
Soft triggers (review then decide):
- Accuracy drop within tolerance but borderline।
- Subgroup performance shift।
- Business KPI dip।
Time-based:
- Bad signal → wait 5 min (transient?) → still bad → rollback।
- Avoid rolling back on flap।
ML-specific challenge — ground truth lag:
- Accuracy needs labels — sometimes hours/days।
- Proxy: prediction distribution drift।
- Don't wait for accuracy — proxy + manual review।
BD context — fraud:
- "Approval rate" sudden change → rollback signal।
- Customer complaint spike → manual review।
Manual override:
- Pause button — keep current % but stop progression।
- Manual rollback — UI button।
মূল উপলব্ধি: Rollback triggers tiered — hard automatic, soft human-review। Speed matters; flap-protection too। Always have manual override।
প্র ০৩"Shadow deployment — practical implementation?"
Shadow = new model receives request copy; response discarded।
Implementation patterns:
(১) API gateway split:
- Gateway sends request to v1 (return) + v2 (log only)।
- Async — v2 result not block response।
- Compare in offline batch।
(২) Service mesh shadow:
- Istio shadow rule — copy traffic, ignore response।
- Built-in, no custom code।
(৩) Backend mirror:
- v1 service after responding, async fire to v2।
- v2 result log to DB।
- Periodic comparison report।
What to compare:
- Prediction distribution (PSI between v1 vs v2)।
- Latency profile।
- Disagreement rate (top-1 prediction differ %)।
- Error rate v2।
Cost:
- 2× compute during shadow।
- 2× external dependency calls (Feature Store)।
- Storage for v2 predictions।
Use case:
- Major architecture change — high uncertainty।
- Schema-shift validation।
- Pre-canary safety net।
BD example:
- Daraz recommendation engine v1→v2: shadow 7 days, compare top-10 disagreement rate।
- If disagreement < threshold → safe to canary।
মূল উপলব্ধি: Shadow = zero-risk validation। Cost duplicate; benefit confidence। Major changes worth।
প্র ০৪"Stateful ML service blue-green challenge?"
Stateful (DB, cache, session) service blue-green tricky।
Challenges:
- DB schema — both versions same DB?
- Cache — invalidate vs duplicate।
- Session affinity — user mid-session, switch breaks।
Patterns:
Backwards-compatible schema:
- v2 schema-additive only (no remove)।
- Both versions same DB simultaneously।
- Cleanup migration after v1 retired।
Feature flag-based:
- Both versions deployed together।
- Flag switch per-user — gradual feature rollout।
- Independent of binary deployment।
Read-replica-based:
- v2 read from replica during warm-up।
- Switch to primary on go-live।
Stateful ML services examples:
- RAG with vector DB (FAISS, Pinecone) — index version।
- Recommendation with user state — session continuity।
BD fintech approach:
- Canary preferred for stateful — gradual transition।
- Blue-green only for stateless components (model serving)।
- Stateful migration separate process।
মূল উপলব্ধি: Pure blue-green stateless service-এর জন্য। Stateful — additive schema, feature flag, gradual। ML serving usually stateless — easy। Vector DB stateful — careful।
অনুশীলন
- Argo Rollouts: Local k8s-এ Argo Rollouts install + sample canary deploy।
Helm install argo-rollouts। Sample yaml apply। UI argocd-rollouts dashboard।
- Analysis: Custom AnalysisTemplate লিখুন — ML-specific (accuracy gate)।
Prometheus query measure model accuracy metric। SuccessCondition based threshold।
- চিন্তা: bKash fraud model v2 deployment plan — canary steps, gates, rollback triggers।
- Shadow 3 days disagreement check।
- Canary 1% (2hr) → 5% (4hr) → 25% (8hr) → 100%।
- Gates: error rate, latency, fraud detection rate, false positive rate।
- Rollback triggers: any gate fail + manual review board approval if borderline।