Canary ও blue-green deployment
এই পাঠে যা শিখবেন
- Canary deployment workflow + monitoring
- Blue-green vs canary trade-offs
- Shadow deployment for ML-specific validation
- Auto-rollback triggers
১ · কেন canary/blue-green
"Replace all pods at once" deployment risky — bad model serve all users immediately.
- Canary: gradual exposure — issue detected early, blast radius small.
- Blue-green: instant rollback — new env fail, switch back instantly.
- Shadow: zero user-impact validation.
২ · Canary deployment
Pattern:
- Deploy v2 alongside v1 (small replica count).
- Route 5% traffic → v2.
- Monitor 30 min: latency, error rate, accuracy.
- If healthy: increase to 25%, 50%, 100%.
- If unhealthy: rollback (route 0% to v2).
৩ · Argo Rollouts canary
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: sentiment-api
spec:
replicas: 10
strategy:
canary:
steps:
- setWeight: 5 # 5% traffic to canary
- pause: { duration: 30m }
- analysis:
templates:
- templateName: success-rate
- templateName: latency-p99
- setWeight: 25
- pause: { duration: 1h }
- analysis: { templates: [{templateName: success-rate}] }
- setWeight: 50
- pause: { duration: 2h }
- setWeight: 100
analysis:
templates:
- templateName: success-rate
startingStep: 1
args: []
trafficRouting:
istio:
virtualService: { name: sentiment-vs }
selector:
matchLabels: { app: sentiment-api }
template:
metadata:
labels: { app: sentiment-api }
spec:
containers:
- name: api
image: registry/sentiment:v2.0
# ... resources, probes
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: { name: success-rate }
spec:
metrics:
- name: success-rate
interval: 5m
successCondition: result[0] >= 0.99
failureLimit: 2
provider:
prometheus:
address: http://prometheus.monitoring:9090
query: |
sum(rate(http_requests_total{job="sentiment-api",status=~"2.."}[5m]))
/
sum(rate(http_requests_total{job="sentiment-api"}[5m]))৪ · Blue-green
Two complete environments, switch traffic atomically.
- Blue: current production, all traffic.
- Green: new version deploy + warm up + smoke test.
- Switch: ingress flip blue → green. Instant.
- Rollback: flip back. Instant.
Pros: instant switch, instant rollback. Cons: 2× resource cost during deployment.
৫ · Shadow deployment (ML-specific)
New model receives copy of production requests; response logged, not returned. Silent compare.
- Production unaffected — zero risk.
- Real production data — best validation.
- Distribution comparison — old vs new prediction.
- Performance compare without committing.
৬ · Auto-rollback triggers
- Error rate spike: > 1% threshold.
- Latency degradation: p99 > SLA.
- Accuracy drop: ground truth available — accuracy gate.
- Drift signal: input/output distribution shift.
- Business KPI: conversion drop, complaint spike.
৭ · Cost of running both
Canary 5% — 1 small extra pod. Blue-green — full duplicate (during deployment). Plan capacity.
ভাবনার প্রশ্ন
প্র ০১"Canary % split — কীভাবে decide?"
Initial weight + step progression — strategy.
Initial weight (5%, 10%, 1%):
- Critical service (payment, fraud): 1% — minimum exposure.
- Standard (recommendation): 5%.
- Experimental (new feature): 10%.
Step progression:
- 5% → 25% → 50% → 100%: aggressive, fast feedback.
- 5% → 10% → 25% → 50% → 100%: cautious, more confidence.
- 1% → 10% → 100%: bimodal — small or full.
Time per step:
- 30 min — fast metric-driven (latency, error).
- 1-2 hours — moderate signal need.
- 24 hours — slow signals (drift, conversion rate).
Statistical sample size:
- 5% × 30 min × 1000 RPS = 90,000 requests — strong signal.
- 1% × 30 min × 100 RPS = 1,800 requests — weak. Increase % or duration.
BD fintech example:
- bKash fraud canary: 1% → 5% → 25% → 100%, each step 2 hours, total 8 hours.
- Conservative due to financial impact.
মূল উপলব্ধি: Canary % depends risk profile + signal volume + business tolerance. Critical service = small initial + slow steps; experimental = aggressive.
প্র ০২"Rollback — কোন metric trigger?"
Rollback decision metric-driven, fast.
Hard triggers (immediate rollback):
- Error rate > 1% (vs < 0.1% baseline).
- p99 latency > 2× baseline.
- Pod crash loop.
Soft triggers (review then decide):
- Accuracy drop within tolerance but borderline.
- Subgroup performance shift.
- Business KPI dip.
Time-based:
- Bad signal → wait 5 min (transient?) → still bad → rollback.
- Avoid rolling back on flap.
ML-specific challenge — ground truth lag:
- Accuracy needs labels — sometimes hours/days.
- Proxy: prediction distribution drift.
- Don't wait for accuracy — proxy + manual review.
BD context — fraud:
- "Approval rate" sudden change → rollback signal.
- Customer complaint spike → manual review.
Manual override:
- Pause button — keep current % but stop progression.
- Manual rollback — UI button.
মূল উপলব্ধি: Rollback triggers tiered — hard automatic, soft human-review. Speed matters; flap-protection too. Always have manual override.
প্র ০৩"Shadow deployment — practical implementation?"
Shadow = new model receives request copy; response discarded.
Implementation patterns:
(১) API gateway split:
- Gateway sends request to v1 (return) + v2 (log only).
- Async — v2 result not block response.
- Compare in offline batch.
(২) Service mesh shadow:
- Istio shadow rule — copy traffic, ignore response.
- Built-in, no custom code.
(৩) Backend mirror:
- v1 service after responding, async fire to v2.
- v2 result log to DB.
- Periodic comparison report.
What to compare:
- Prediction distribution (PSI between v1 vs v2).
- Latency profile.
- Disagreement rate (top-1 prediction differ %).
- Error rate v2.
Cost:
- 2× compute during shadow.
- 2× external dependency calls (Feature Store).
- Storage for v2 predictions.
Use case:
- Major architecture change — high uncertainty.
- Schema-shift validation.
- Pre-canary safety net.
BD example:
- Daraz recommendation engine v1→v2: shadow 7 days, compare top-10 disagreement rate.
- If disagreement < threshold → safe to canary.
মূল উপলব্ধি: Shadow = zero-risk validation. Cost duplicate; benefit confidence. Major changes worth.
প্র ০৪"Stateful ML service blue-green challenge?"
Stateful (DB, cache, session) service blue-green tricky.
Challenges:
- DB schema — both versions same DB?
- Cache — invalidate vs duplicate.
- Session affinity — user mid-session, switch breaks.
Patterns:
Backwards-compatible schema:
- v2 schema-additive only (no remove).
- Both versions same DB simultaneously.
- Cleanup migration after v1 retired.
Feature flag-based:
- Both versions deployed together.
- Flag switch per-user — gradual feature rollout.
- Independent of binary deployment.
Read-replica-based:
- v2 read from replica during warm-up.
- Switch to primary on go-live.
Stateful ML services examples:
- RAG with vector DB (FAISS, Pinecone) — index version.
- Recommendation with user state — session continuity.
BD fintech approach:
- Canary preferred for stateful — gradual transition.
- Blue-green only for stateless components (model serving).
- Stateful migration separate process.
মূল উপলব্ধি: Pure blue-green stateless service-এর জন্য। Stateful — additive schema, feature flag, gradual. ML serving usually stateless — easy. Vector DB stateful — careful.
অনুশীলন
- Argo Rollouts: Local k8s-এ Argo Rollouts install + sample canary deploy.
Helm install argo-rollouts. Sample yaml apply. UI argocd-rollouts dashboard.
- Analysis: Custom AnalysisTemplate লিখুন — ML-specific (accuracy gate).
Prometheus query measure model accuracy metric. SuccessCondition based threshold.
- চিন্তা: bKash fraud model v2 deployment plan — canary steps, gates, rollback triggers.
- Shadow 3 days disagreement check.
- Canary 1% (2hr) → 5% (4hr) → 25% (8hr) → 100%.
- Gates: error rate, latency, fraud detection rate, false positive rate.
- Rollback triggers: any gate fail + manual review board approval if borderline.