পাঠ ৭ · ৩৩-এর মধ্যে · মডিউল ১

Home / AI Courses / MLOps / Kubernetes basics

Kubernetes পরিচিতি

Kubernetes basics for ML

৭ মিনিট পড়া মধ্য · Intermediate YAML

এই পাঠে যা শিখবেন

K8s-এর core resource — Pod, Deployment, Service, Ingress
ML model serving-এর একটি minimal manifest
GPU scheduling — node selector, taint/toleration
কখন K8s overkill, কখন essential

১ · Kubernetes কী, কেন

একটি Docker container চালু রাখা সহজ। ১০০টি container, ১০টি machine, traffic spike-এ auto-scale, একটা machine ফেইল করলে নতুন place-এ start — এই সব hand-এ manage করা অসম্ভব। KubernetesKubernetes (K8s)Google-এর Borg-inspired open-source container orchestrator. কোন container কোন node-এ চলবে, কতগুলো replica, কীভাবে network expose — declaratively manage. (K8s) এই job করে।

Auto-scaling: traffic বাড়লে replica বাড়ায়।
Self-healing: container crash → restart, node fail → অন্য node-এ schedule.
Rolling update: old version চালু থাকতে থাকতে new version deploy.
Service discovery: "model-serving" name → IP automatically.
Resource scheduling: "GPU দরকার" → GPU-node-এ pod place.

২ · Core building blocks

৪টি basic resource

Pod = ১+ container একসাথে (smallest unit).
Deployment = N replica Pod + rolling update strategy.
Service = stable network endpoint (internal IP).
Ingress = external HTTP traffic routing.

৩ · একটি minimal model-serving manifest

YAML · Deployment + Service

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sentiment-api
  labels:
    app: sentiment-api
    model-version: v1.3.0
spec:
  replicas: 3
  selector:
    matchLabels:
      app: sentiment-api
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  template:
    metadata:
      labels:
        app: sentiment-api
        model-version: v1.3.0
    spec:
      containers:
        - name: api
          image: registry.example.com/sentiment-api:v1.3.0
          ports:
            - containerPort: 8000
          env:
            - name: MODEL_VERSION
              value: "v1.3.0"
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
            limits:
              cpu: "2"
              memory: "4Gi"
          readinessProbe:
            httpGet: { path: /healthz, port: 8000 }
            initialDelaySeconds: 5
            periodSeconds: 5
          livenessProbe:
            httpGet: { path: /healthz, port: 8000 }
            initialDelaySeconds: 30
            periodSeconds: 15
---
apiVersion: v1
kind: Service
metadata:
  name: sentiment-api
spec:
  selector:
    app: sentiment-api
  ports:
    - port: 80
      targetPort: 8000
  type: ClusterIP

গুরুত্বপূর্ণ patterns: (১) replicas: 3 — fault tolerance, (২) RollingUpdate maxUnavailable=0 — zero-downtime deploy, (৩) resource requests vs limits, (৪) readiness vs liveness probe, (৫) version label — A/B test ও rollback-এ কাজে আসে।

৪ · GPU scheduling

GPU node-এ pod place করতে — resource request + node selector.

YAML · GPU pod

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-inference
spec:
  replicas: 2
  selector:
    matchLabels: { app: llm-inference }
  template:
    metadata:
      labels: { app: llm-inference }
    spec:
      nodeSelector:
        nvidia.com/gpu.product: "NVIDIA-A100"
      tolerations:
        - key: "nvidia.com/gpu"
          operator: "Exists"
          effect: "NoSchedule"
      containers:
        - name: llm
          image: registry.example.com/llm-server:v0.4
          resources:
            limits:
              nvidia.com/gpu: 1     # ⭐ GPU request
              cpu: "4"
              memory: "16Gi"

Cluster-এ NVIDIA GPU Operator install থাকতে হবে। তাহলে nvidia.com/gpu resource type recognize হয়। Node selector specific GPU type-এ targeting (A100 vs T4).

৫ · ML-native add-ons

NVIDIA GPU Operator: driver + plugin + monitoring auto-install.
Kubeflow: end-to-end ML platform on K8s — pipelines, notebooks, serving.
KServe (formerly KFServing): declarative model serving — autoscaling, canary, multi-model.
Seldon Core: alternative serving platform, advanced inference graphs.
Argo Workflows: K8s-native DAG orchestration — training pipelines.
Volcano / Kueue: batch/training job scheduler — gang scheduling, fair share.

৬ · কখন K8s overkill

একটি model, low traffic — single VM + Docker compose যথেষ্ট।
Team-এ K8s expertise নেই — operational burden উচ্চ।
Stateful training data না থাকলে — managed serverless (Cloud Run, Lambda) সহজ।

৭ · কখন K8s essential

Multiple model + multi-team.
Auto-scaling + traffic-spike handling.
GPU pool sharing across teams.
ML platform team building self-service infrastructure.
Hybrid cloud / multi-region deployment.

Ingress external traffic accepts → Service load balance করে → Deployment-এর Pods (CPU বা GPU node-এ) chose. Node selector + GPU resource — proper place ঠিক করে।

Bangladeshi context-এ — managed K8s (GKE Mumbai, EKS Mumbai, AKS Singapore) সবচেয়ে সহজ। Self-hosted "kubeadm" cluster small team-এ overhead বেশি। তবে compliance reasons-এ on-prem necessary হলে — Rancher / k3s lightweight option.

ভাবনার প্রশ্ন

প্র ০১ একটি ML team K8s adopt করছে। প্রথম ৩০ দিনে কী কী topic master করা প্রয়োজন? কোনগুলো পরে রাখা যায়?

K8s বিশাল — সব একসাথে শেখা impossible. ML team-এর জন্য priority থাকা চাই।

প্রথম ৩০ দিন — must master:

kubectl basics: apply, get, describe, logs, exec, port-forward. প্রতিদিনের commands.
Pod, Deployment, Service: ৯০% ML serving এই তিনটির উপর।
ConfigMap + Secret: environment configuration externalize.
Resource request/limit: কোন pod কত CPU/RAM/GPU. OOM debug essential.
Probes (readiness/liveness): ML model load হতে সময় লাগে — wrong probe = constant restart.

৩০-৬০ দিন — important next:

Ingress + TLS: external traffic, HTTPS.
HPA (Horizontal Pod Autoscaler): traffic-based scaling.
PVC (PersistentVolumeClaim): model artifact storage.
Namespaces + RBAC: team isolation.
Helm: reusable manifest packages.

৬০-৯০ দিন — advanced:

Network policy, service mesh (Istio basic).
Operator pattern (ArgoCD, Flux for GitOps).
StatefulSet (rare in ML but useful for distributed training).
Custom Resource Definitions (KServe, Kubeflow uses).
Cluster autoscaler tuning.

"কখনো না" বা delayed:

Cluster admin level: kubeadm, etcd backup — managed K8s হলে cloud provider-এর কাজ।
Custom CNI tuning — usually default fine.
Pod security policies — modern alternatives (Pod Security Standards) আছে।

ML-specific learning order:

Plain Deployment first. তারপর GPU resource. তারপর KServe বা Kubeflow.
"Kubeflow first" trap — অনেকে পড়ে। K8s না বুঝে Kubeflow debug অসম্ভব।

Practical learning resource:

kubectl cheatsheet — desk-এ রাখুন।
Local cluster: minikube, kind, k3d — laptop-এ K8s.
"Kubernetes the hard way" (Kelsey Hightower) — eventually, না initially.
BD context — Bangladesh-এ K8s meetup এখন active, networking opportunity.

মূল উপলব্ধি: K8s-এর scope বিশাল কিন্তু ML-specific use case-এ ৪০% surface এই সম্পূর্ণ। Pod/Deployment/Service/probe — এই ৪টি deeply জানলে ৭০% job ঢেকে যাবে। Premature deep dive (Operator, CRD) — actual problem আসার আগে অপ্রয়োজনীয়।

প্র ০২ "Managed K8s (GKE/EKS/AKS) vs self-hosted (kubeadm)" — Bangladesh-এর ML team-এর জন্য decision factors কী?

এই decision strategic — operational burden, cost, ও compliance ত্রিভুজ।

Managed K8s — advantages:

Control plane managed by cloud — etcd backup, cert rotation, K8s upgrade auto.
Faster setup (~৩০ মিনিটে cluster ready).
GPU node pool prebuilt.
Cloud integration (LoadBalancer, storage class).

Managed K8s — disadvantages:

Cost — control plane ~$৭৫/month + node cost.
Vendor lock-in — gentle but real.
Specific cloud-এ; multi-cloud কঠিন।

Self-hosted — advantages:

Cost — only hardware/VM cost.
Full control — kernel, network, storage সব tweak.
On-prem compliance — data leave country না।

Self-hosted — disadvantages:

Operational team দরকার — ১-২ dedicated K8s engineer.
Upgrade painful — major version transition.
Security patching responsibility.
Storage layer (Ceph, longhorn) complex.

Bangladesh-specific factors:

Latency: nearest cloud region — Mumbai (AWS, GCP), Singapore (AWS, Azure). ~৪০-৬০ ms RTT. User-facing API-এ matters.
Bandwidth cost: egress $/GB — managed cloud-এ careful.
Regulation: Bangladesh Bank guideline — financial data on-prem preference. Hybrid common (compute on-prem, dev tools cloud).
Power/cooling: on-prem-এ data center quality varies; large GPU cluster heat management.
Talent availability: K8s-experienced engineer Bangladesh-এ scarce; managed-এ talent gap কম।

Hybrid pattern (common):

Production serving on-prem (data privacy).
Training jobs on managed cloud (burst GPU).
Staging/dev cloud (rapid iteration).
Tooling (MLflow, registry) cloud (ease).

Decision tree:

Team < ৫ engineer → managed. Self-host operational tax too high.
Team ৫-২০ + 'devops' competence → either, lean managed for ML focus.
Team ২০+, 'platform' team exists, on-prem mandate → self-hosted with hybrid bursting.
Strict compliance + data residency → on-prem with disaster recovery cloud.

Cost comparison (rough, USD/month for ৫ ML services):

Managed (GKE Mumbai): control plane $৭৫ + 5 nodes × $২০০ = $১,০৭৫।
Self-hosted (3 VPS Linode): $৩×$১০০ = $৩০০ + ops engineer salary $২,৫০০ = $২,৮০০।
Self-hosted only cheaper at scale (১০+ services).

মূল উপলব্ধি: Bangladesh-এর majority ML team-এর জন্য managed K8s — সঠিক choice. On-prem-এর justification compliance/data residency-এ। Hybrid practical realistically — control over critical part, agility over rest.

প্র ০৩ "একটি training job ১২ ঘণ্টা চলে। Node ফেইল হলে কী ঘটে — কীভাবে handle?"

Long-running training-এর fault tolerance — distributed ML-এর core challenge. K8s built-in solution incomplete.

Default behavior:

Pod = single training job. Node fail → Pod terminated.
Deployment = stateless replica. Training-এ inappropriate.
Job/CronJob — limited retry; checkpoint-aware না।

সমস্যা specifics:

১২ ঘণ্টায় ৫০% trained, GPU spot interruption — সব lost.
Multi-GPU training-এ একটি pod fail = whole job fail (gang scheduling).
Retry-এ scratch থেকে শুরু — wasted compute.

Solution layers:

(১) Application-level checkpointing:

Training script every N steps checkpoint save (S3, PVC).
Resume logic: latest checkpoint detect, load.
PyTorch Lightning, Hugging Face Trainer — auto-checkpoint built-in.

(২) K8s retry:

Job backoffLimit: 3, activeDeadlineSeconds.
Pod restart-এ checkpoint থেকে resume.

(৩) Volcano / Kueue scheduler:

K8s-native batch scheduler.
Gang scheduling (all pods together or none).
Priority queue, preemption-safe.
Distributed training (PyTorch DDP, Horovod) friendly.

(৪) Kubeflow Training Operator:

PyTorchJob, TFJob CRD.
Multi-pod coordination, auto-restart.
Checkpoint integration with model registry.

(৫) Spot/preemptible GPU strategy:

Spot ৫০-৭০% cheaper but interruptible.
Frequent checkpointing (every ১৫ মিনিট)।
Mixed pool: critical jobs on-demand, dev jobs spot.

BD-specific consideration:

On-prem GPU cluster ক্ষেত্রে power outage (load shedding) frequent. UPS-এর backup limited.
Dataset large + training long → break চলাকালীন cost considerable.
Recommendation: critical training cloud-এ (stable infra), experimentation on-prem.

Common mistake:

"Replicas: 3" Deployment-এ training launch — multiple parallel runs, conflicting checkpoint write.
Job-এ completions: 1 + checkpoint logic correct way.

Distributed training fault tolerance:

One pod fail → entire ring break in DDP.
Elastic training (PyTorch torchrun --nnodes dynamic) — partial recovery.
Worker failure: skip-and-restart, not entire job restart.

মূল উপলব্ধি: Long training fault tolerance K8s alone solve করে না — application + scheduler + storage সম্মিলিত responsibility. Checkpointing application's responsibility; scheduler retry; storage durable. তিনটি একসাথে কাজ করলে — ১২-ঘণ্টা training-এ node failure recoverable.

প্র ০৪ "GPU sharing across teams" — একটি GPU pool, ৫ ML team. কীভাবে fair allocation ও over-utilization avoid?

GPU expensive — sharing essential কিন্তু painful. Multi-tenancy K8s-এর সাথে নানা strategy.

চ্যালেঞ্জ:

GPU exclusive resource — fractional sharing tricky.
Memory bound jobs ও compute bound jobs — different profile.
Long-running (training) ও short-running (inference) — scheduling conflict.

Strategy ১ — Whole GPU per pod (simplest):

nvidia.com/gpu: 1 — exclusive use.
Pros: clean, no interference.
Cons: small inference পুরো A100 — wasteful.

Strategy ২ — MIG (Multi-Instance GPU, A100/H100):

A100 hardware-level partitions — 7 isolated instances.
K8s-এ MIG-aware scheduling.
Pros: hardware isolation, predictable performance.
Cons: A100/H100 only; partition fixed at boot.

Strategy ৩ — Time-sharing (NVIDIA GPU Operator):

Multiple pods share GPU, time-slice.
Pros: flexible, smaller GPU-ও sharable.
Cons: noisy neighbor, latency unpredictable.

Strategy ৪ — Namespace ResourceQuota:

Per-team GPU budget — namespace quota.
Team-A: 4 GPUs, Team-B: 2 GPUs.
Hard cap; over-request rejected.

Strategy ৫ — Priority + preemption:

Production inference high priority.
Dev/research lower; preemptable.
Volcano scheduler-এ fairshare policy.

Practical multi-team setup:

Production namespace: dedicated GPU quota, cannot preempt.
Training namespace: scheduled queue, shared pool.
Research namespace: low priority, preemptable, spot-like.
Inference autoscaler: small per-pod (T4 or MIG slice), many replicas.

Cost attribution:

Label every pod with team/project name.
kube-state-metrics + Prometheus → GPU-hour per team.
Monthly chargeback report.

"Idle GPU = wasted GPU":

Notebook server pinned GPU all day, used 1 hour — common waste.
Auto-shutdown idle notebooks (Kubeflow Notebooks-এ feature).
"Use it or lose it" policy.

Bangladesh context note:

GPU expensive ($1.5-3/hour A100). Sharing critical.
Local cloud (BDIX, BD-tier providers) GPU offerings limited.
Hybrid common: short jobs cloud, long-batch on-prem.

মূল উপলব্ধি: Multi-team GPU sharing — pure technical না, organizational governance-ও। Quota + priority + chargeback combined — fair pattern. MIG + autoscaling + idle shutdown — efficiency. Single tool magic না — combination.

অনুশীলন

Manifest: উপরের sentiment-api manifest নিয়ে — replicas বাড়িয়ে ৫ করুন, একটি ConfigMap যোগ করুন (model URL store), ও env-এ ব্যবহার করুন।
```
apiVersion: v1
kind: ConfigMap
metadata: { name: sentiment-config }
data:
  MODEL_URL: "s3://my-bucket/models/sentiment-v1.3.0.joblib"
---
# in container env:
envFrom:
  - configMapRef: { name: sentiment-config }
```
চিন্তা: কখন আপনি একটি ML team-কে K8s-এ যেতে পরামর্শ দেবেন না? ৩টি specific scenario.
- একটি model, monthly batch — Cloud Function/Lambda যথেষ্ট।
- Team-এ DevOps expertise নেই + dedicated infra-এর budget নেই।
- Ultra-low-latency edge — K8s overhead ms matters.
Diagnose: একটি ML pod CrashLoopBackOff state-এ। কী ৩টি command চালাবেন debug-এ?
- kubectl describe pod <name> — events, scheduling problem.
- kubectl logs <name> --previous — previous crash log.
- kubectl exec -it <name> -- /bin/bash — interactive (যদি pod চলতে থাকে)। OR check resource limits/probes.