পাঠ ৭ · ৩৩-এর মধ্যে · মডিউল ১
Home / AI Courses / MLOps / Kubernetes basics

Kubernetes পরিচিতি

Kubernetes basics for ML
৭ মিনিট পড়া মধ্য · Intermediate YAML

এই পাঠে যা শিখবেন

  • K8s-এর core resource — Pod, Deployment, Service, Ingress
  • ML model serving-এর একটি minimal manifest
  • GPU scheduling — node selector, taint/toleration
  • কখন K8s overkill, কখন essential

১ · Kubernetes কী, কেন

একটি Docker container চালু রাখা সহজ। ১০০টি container, ১০টি machine, traffic spike-এ auto-scale, একটা machine ফেইল করলে নতুন place-এ start — এই সব hand-এ manage করা অসম্ভব। KubernetesKubernetes (K8s)Google-এর Borg-inspired open-source container orchestrator। কোন container কোন node-এ চলবে, কতগুলো replica, কীভাবে network expose — declaratively manage। (K8s) এই job করে।

  • Auto-scaling: traffic বাড়লে replica বাড়ায়।
  • Self-healing: container crash → restart, node fail → অন্য node-এ schedule।
  • Rolling update: old version চালু থাকতে থাকতে new version deploy।
  • Service discovery: "model-serving" name → IP automatically।
  • Resource scheduling: "GPU দরকার" → GPU-node-এ pod place।

২ · Core building blocks

৪টি basic resource

Pod = ১+ container একসাথে (smallest unit)।
Deployment = N replica Pod + rolling update strategy।
Service = stable network endpoint (internal IP)।
Ingress = external HTTP traffic routing।

৩ · একটি minimal model-serving manifest

YAML · Deployment + Service 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sentiment-api
  labels:
    app: sentiment-api
    model-version: v1.3.0
spec:
  replicas: 3
  selector:
    matchLabels:
      app: sentiment-api
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  template:
    metadata:
      labels:
        app: sentiment-api
        model-version: v1.3.0
    spec:
      containers:
        - name: api
          image: registry.example.com/sentiment-api:v1.3.0
          ports:
            - containerPort: 8000
          env:
            - name: MODEL_VERSION
              value: "v1.3.0"
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
            limits:
              cpu: "2"
              memory: "4Gi"
          readinessProbe:
            httpGet: { path: /healthz, port: 8000 }
            initialDelaySeconds: 5
            periodSeconds: 5
          livenessProbe:
            httpGet: { path: /healthz, port: 8000 }
            initialDelaySeconds: 30
            periodSeconds: 15
---
apiVersion: v1
kind: Service
metadata:
  name: sentiment-api
spec:
  selector:
    app: sentiment-api
  ports:
    - port: 80
      targetPort: 8000
  type: ClusterIP

    


    

      গুরুত্বপূর্ণ patterns: (১) replicas: 3 — fault tolerance, (২) RollingUpdate maxUnavailable=0 — zero-downtime deploy, (৩) resource requests vs limits, (৪) readiness vs liveness probe, (৫) version label — A/B test ও rollback-এ কাজে আসে।
    


    
৪ · GPU scheduling


    

      GPU node-এ pod place করতে — resource request + node selector।
    


    

      

        
        YAML · GPU pod
      

      
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-inference
spec:
  replicas: 2
  selector:
    matchLabels: { app: llm-inference }
  template:
    metadata:
      labels: { app: llm-inference }
    spec:
      nodeSelector:
        nvidia.com/gpu.product: "NVIDIA-A100"
      tolerations:
        - key: "nvidia.com/gpu"
          operator: "Exists"
          effect: "NoSchedule"
      containers:
        - name: llm
          image: registry.example.com/llm-server:v0.4
          resources:
            limits:
              nvidia.com/gpu: 1     # ⭐ GPU request
              cpu: "4"
              memory: "16Gi"

      

        
        
      

      

    


    

      Cluster-এ NVIDIA GPU Operator install থাকতে হবে। তাহলে nvidia.com/gpu resource type recognize হয়। Node selector specific GPU type-এ targeting (A100 vs T4)।
    


    
৫ · ML-native add-ons


    
    
      
  • NVIDIA GPU Operator: driver + plugin + monitoring auto-install।
    • 
      
  • Kubeflow: end-to-end ML platform on K8s — pipelines, notebooks, serving।
    • 
      
  • KServe (formerly KFServing): declarative model serving — autoscaling, canary, multi-model।
    • 
      
  • Seldon Core: alternative serving platform, advanced inference graphs।
    • 
      
  • Argo Workflows: K8s-native DAG orchestration — training pipelines।
    • 
      
  • Volcano / Kueue: batch/training job scheduler — gang scheduling, fair share।
    • 
    


    
৬ · কখন K8s overkill


    
    
      
  • একটি model, low traffic — single VM + Docker compose যথেষ্ট।
    • 
      
  • Team-এ K8s expertise নেই — operational burden উচ্চ।
    • 
      
  • Stateful training data না থাকলে — managed serverless (Cloud Run, Lambda) সহজ।
    • 
    


    
৭ · কখন K8s essential


    
    
      
  • Multiple model + multi-team।
    • 
      
  • Auto-scaling + traffic-spike handling।
    • 
      
  • GPU pool sharing across teams।
    • 
      
  • ML platform team building self-service infrastructure।
    • 
      
  • Hybrid cloud / multi-region deployment।
    • 
    


    

      
        
          
            
          
        

        Kubernetes ML cluster — typical layout
        Ingress → Service → Deployment(Pods) on Nodes

        
        
        K8s Cluster

        
        
        Ingress

        
        
        Service

        
        
        Node 1 (CPU)
        
        Pod-A
        
        Pod-B
        
        Pod-C (sentiment-api)

        
        
        Node 2 (GPU · A100 ×2)
        
        Pod-D (llm-inference)
        nvidia.com/gpu: 1
        
        Pod-E (llm-inference, GPU 2)

        
        
        
        
      
      
Ingress external traffic accepts → Service load balance করে → Deployment-এর Pods (CPU বা GPU node-এ) chose। Node selector + GPU resource — proper place ঠিক করে।

    


    

      Bangladeshi context-এ — managed K8s (GKE Mumbai, EKS Mumbai, AKS Singapore) সবচেয়ে সহজ। Self-hosted "kubeadm" cluster small team-এ overhead বেশি। তবে compliance reasons-এ on-prem necessary হলে — Rancher / k3s lightweight option।
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      
        প্র ০১
        একটি ML team K8s adopt করছে। প্রথম ৩০ দিনে কী কী topic master করা প্রয়োজন? কোনগুলো পরে রাখা যায়?
      
      

        

          K8s বিশাল — সব একসাথে শেখা impossible। ML team-এর জন্য priority থাকা চাই।
        

        
প্রথম ৩০ দিন — must master:

        
    
          
  • kubectl basics: apply, get, describe, logs, exec, port-forward। প্রতিদিনের commands।
    • 
          
  • Pod, Deployment, Service: ৯০% ML serving এই তিনটির উপর।
    • 
          
  • ConfigMap + Secret: environment configuration externalize।
    • 
          
  • Resource request/limit: কোন pod কত CPU/RAM/GPU। OOM debug essential।
    • 
          
  • Probes (readiness/liveness): ML model load হতে সময় লাগে — wrong probe = constant restart।
    • 
        

        
৩০-৬০ দিন — important next:

        
    
          
  • Ingress + TLS: external traffic, HTTPS।
    • 
          
  • HPA (Horizontal Pod Autoscaler): traffic-based scaling।
    • 
          
  • PVC (PersistentVolumeClaim): model artifact storage।
    • 
          
  • Namespaces + RBAC: team isolation।
    • 
          
  • Helm: reusable manifest packages।
    • 
        

        
৬০-৯০ দিন — advanced:

        
    
          
  • Network policy, service mesh (Istio basic)।
    • 
          
  • Operator pattern (ArgoCD, Flux for GitOps)।
    • 
          
  • StatefulSet (rare in ML but useful for distributed training)।
    • 
          
  • Custom Resource Definitions (KServe, Kubeflow uses)।
    • 
          
  • Cluster autoscaler tuning।
    • 
        

        
"কখনো না" বা delayed:

        
    
          
  • Cluster admin level: kubeadm, etcd backup — managed K8s হলে cloud provider-এর কাজ।
    • 
          
  • Custom CNI tuning — usually default fine।
    • 
          
  • Pod security policies — modern alternatives (Pod Security Standards) আছে।
    • 
        

        
ML-specific learning order:

        
    
          
  • Plain Deployment first। তারপর GPU resource। তারপর KServe বা Kubeflow।
    • 
          
  • "Kubeflow first" trap — অনেকে পড়ে। K8s না বুঝে Kubeflow debug অসম্ভব।
    • 
        

        
Practical learning resource:

        
    
          
  • kubectl cheatsheet — desk-এ রাখুন।
    • 
          
  • Local cluster: minikube, kind, k3d — laptop-এ K8s।
    • 
          
  • "Kubernetes the hard way" (Kelsey Hightower) — eventually, না initially।
    • 
          
  • BD context — Bangladesh-এ K8s meetup এখন active, networking opportunity।
    • 
        

        

          মূল উপলব্ধি: K8s-এর scope বিশাল কিন্তু ML-specific use case-এ ৪০% surface এই সম্পূর্ণ। Pod/Deployment/Service/probe — এই ৪টি deeply জানলে ৭০% job ঢেকে যাবে। Premature deep dive (Operator, CRD) — actual problem আসার আগে অপ্রয়োজনীয়।
        

      

    


    

      
        প্র ০২
        "Managed K8s (GKE/EKS/AKS) vs self-hosted (kubeadm)" — Bangladesh-এর ML team-এর জন্য decision factors কী?
      
      

        

          এই decision strategic — operational burden, cost, ও compliance ত্রিভুজ।
        

        
Managed K8s — advantages:

        
    
          
  • Control plane managed by cloud — etcd backup, cert rotation, K8s upgrade auto।
    • 
          
  • Faster setup (~৩০ মিনিটে cluster ready)।
    • 
          
  • GPU node pool prebuilt।
    • 
          
  • Cloud integration (LoadBalancer, storage class)।
    • 
        

        
Managed K8s — disadvantages:

        
    
          
  • Cost — control plane ~$৭৫/month + node cost।
    • 
          
  • Vendor lock-in — gentle but real।
    • 
          
  • Specific cloud-এ; multi-cloud কঠিন।
    • 
        

        
Self-hosted — advantages:

        
    
          
  • Cost — only hardware/VM cost।
    • 
          
  • Full control — kernel, network, storage সব tweak।
    • 
          
  • On-prem compliance — data leave country না।
    • 
        

        
Self-hosted — disadvantages:

        
    
          
  • Operational team দরকার — ১-২ dedicated K8s engineer।
    • 
          
  • Upgrade painful — major version transition।
    • 
          
  • Security patching responsibility।
    • 
          
  • Storage layer (Ceph, longhorn) complex।
    • 
        

        
Bangladesh-specific factors:

        
    
          
  • Latency: nearest cloud region — Mumbai (AWS, GCP), Singapore (AWS, Azure)। ~৪০-৬০ ms RTT। User-facing API-এ matters।
    • 
          
  • Bandwidth cost: egress $/GB — managed cloud-এ careful।
    • 
          
  • Regulation: Bangladesh Bank guideline — financial data on-prem preference। Hybrid common (compute on-prem, dev tools cloud)।
    • 
          
  • Power/cooling: on-prem-এ data center quality varies; large GPU cluster heat management।
    • 
          
  • Talent availability: K8s-experienced engineer Bangladesh-এ scarce; managed-এ talent gap কম।
    • 
        

        
Hybrid pattern (common):

        
    
          
  • Production serving on-prem (data privacy)।
    • 
          
  • Training jobs on managed cloud (burst GPU)।
    • 
          
  • Staging/dev cloud (rapid iteration)।
    • 
          
  • Tooling (MLflow, registry) cloud (ease)।
    • 
        

        
Decision tree:

        
    
          
  • Team < ৫ engineer → managed। Self-host operational tax too high।
    • 
          
  • Team ৫-২০ + 'devops' competence → either, lean managed for ML focus।
    • 
          
  • Team ২০+, 'platform' team exists, on-prem mandate → self-hosted with hybrid bursting।
    • 
          
  • Strict compliance + data residency → on-prem with disaster recovery cloud।
    • 
        

        
Cost comparison (rough, USD/month for ৫ ML services):

        
    
          
  • Managed (GKE Mumbai): control plane $৭৫ + 5 nodes × $২০০ = $১,০৭৫।
    • 
          
  • Self-hosted (3 VPS Linode): $৩×$১০০ = $৩০০ + ops engineer salary $২,৫০০ = $২,৮০০।
    • 
          
  • Self-hosted only cheaper at scale (১০+ services)।
    • 
        

        

          মূল উপলব্ধি: Bangladesh-এর majority ML team-এর জন্য managed K8s — সঠিক choice। On-prem-এর justification compliance/data residency-এ। Hybrid practical realistically — control over critical part, agility over rest।
        

      

    


    

      
        প্র ০৩
        "একটি training job ১২ ঘণ্টা চলে। Node ফেইল হলে কী ঘটে — কীভাবে handle?"
      
      

        

          Long-running training-এর fault tolerance — distributed ML-এর core challenge। K8s built-in solution incomplete।
        

        
Default behavior:

        
    
          
  • Pod = single training job। Node fail → Pod terminated।
    • 
          
  • Deployment = stateless replica। Training-এ inappropriate।
    • 
          
  • Job/CronJob — limited retry; checkpoint-aware না।
    • 
        

        
সমস্যা specifics:

        
    
          
  • ১২ ঘণ্টায় ৫০% trained, GPU spot interruption — সব lost।
    • 
          
  • Multi-GPU training-এ একটি pod fail = whole job fail (gang scheduling)।
    • 
          
  • Retry-এ scratch থেকে শুরু — wasted compute।
    • 
        

        
Solution layers:

        
(১) Application-level checkpointing:

        
    
          
  • Training script every N steps checkpoint save (S3, PVC)।
    • 
          
  • Resume logic: latest checkpoint detect, load।
    • 
          
  • PyTorch Lightning, Hugging Face Trainer — auto-checkpoint built-in।
    • 
        

        
(২) K8s retry:

        
    
          
  • Job backoffLimit: 3, activeDeadlineSeconds।
    • 
          
  • Pod restart-এ checkpoint থেকে resume।
    • 
        

        
(৩) Volcano / Kueue scheduler:

        
    
          
  • K8s-native batch scheduler।
    • 
          
  • Gang scheduling (all pods together or none)।
    • 
          
  • Priority queue, preemption-safe।
    • 
          
  • Distributed training (PyTorch DDP, Horovod) friendly।
    • 
        

        
(৪) Kubeflow Training Operator:

        
    
          
  • PyTorchJob, TFJob CRD।
    • 
          
  • Multi-pod coordination, auto-restart।
    • 
          
  • Checkpoint integration with model registry।
    • 
        

        
(৫) Spot/preemptible GPU strategy:

        
    
          
  • Spot ৫০-৭০% cheaper but interruptible।
    • 
          
  • Frequent checkpointing (every ১৫ মিনিট)।
    • 
          
  • Mixed pool: critical jobs on-demand, dev jobs spot।
    • 
        

        
BD-specific consideration:

        
    
          
  • On-prem GPU cluster ক্ষেত্রে power outage (load shedding) frequent। UPS-এর backup limited।
    • 
          
  • Dataset large + training long → break চলাকালীন cost considerable।
    • 
          
  • Recommendation: critical training cloud-এ (stable infra), experimentation on-prem।
    • 
        

        
Common mistake:

        
    
          
  • "Replicas: 3" Deployment-এ training launch — multiple parallel runs, conflicting checkpoint write।
    • 
          
  • Job-এ completions: 1 + checkpoint logic correct way।
    • 
        

        
Distributed training fault tolerance:

        
    
          
  • One pod fail → entire ring break in DDP।
    • 
          
  • Elastic training (PyTorch torchrun --nnodes dynamic) — partial recovery।
    • 
          
  • Worker failure: skip-and-restart, not entire job restart।
    • 
        

        

          মূল উপলব্ধি: Long training fault tolerance K8s alone solve করে না — application + scheduler + storage সম্মিলিত responsibility। Checkpointing application's responsibility; scheduler retry; storage durable। তিনটি একসাথে কাজ করলে — ১২-ঘণ্টা training-এ node failure recoverable।
        

      

    


    

      
        প্র ০৪
        "GPU sharing across teams" — একটি GPU pool, ৫ ML team। কীভাবে fair allocation ও over-utilization avoid?
      
      

        

          GPU expensive — sharing essential কিন্তু painful। Multi-tenancy K8s-এর সাথে নানা strategy।
        

        
চ্যালেঞ্জ:

        
    
          
  • GPU exclusive resource — fractional sharing tricky।
    • 
          
  • Memory bound jobs ও compute bound jobs — different profile।
    • 
          
  • Long-running (training) ও short-running (inference) — scheduling conflict।
    • 
        

        
Strategy ১ — Whole GPU per pod (simplest):

        
    
          
  • nvidia.com/gpu: 1 — exclusive use।
    • 
          
  • Pros: clean, no interference।
    • 
          
  • Cons: small inference পুরো A100 — wasteful।
    • 
        

        
Strategy ২ — MIG (Multi-Instance GPU, A100/H100):

        
    
          
  • A100 hardware-level partitions — 7 isolated instances।
    • 
          
  • K8s-এ MIG-aware scheduling।
    • 
          
  • Pros: hardware isolation, predictable performance।
    • 
          
  • Cons: A100/H100 only; partition fixed at boot।
    • 
        

        
Strategy ৩ — Time-sharing (NVIDIA GPU Operator):

        
    
          
  • Multiple pods share GPU, time-slice।
    • 
          
  • Pros: flexible, smaller GPU-ও sharable।
    • 
          
  • Cons: noisy neighbor, latency unpredictable।
    • 
        

        
Strategy ৪ — Namespace ResourceQuota:

        
    
          
  • Per-team GPU budget — namespace quota।
    • 
          
  • Team-A: 4 GPUs, Team-B: 2 GPUs।
    • 
          
  • Hard cap; over-request rejected।
    • 
        

        
Strategy ৫ — Priority + preemption:

        
    
          
  • Production inference high priority।
    • 
          
  • Dev/research lower; preemptable।
    • 
          
  • Volcano scheduler-এ fairshare policy।
    • 
        

        
Practical multi-team setup:

        
    
          
  • Production namespace: dedicated GPU quota, cannot preempt।
    • 
          
  • Training namespace: scheduled queue, shared pool।
    • 
          
  • Research namespace: low priority, preemptable, spot-like।
    • 
          
  • Inference autoscaler: small per-pod (T4 or MIG slice), many replicas।
    • 
        

        
Cost attribution:

        
    
          
  • Label every pod with team/project name।
    • 
          
  • kube-state-metrics + Prometheus → GPU-hour per team।
    • 
          
  • Monthly chargeback report।
    • 
        

        
"Idle GPU = wasted GPU":

        
    
          
  • Notebook server pinned GPU all day, used 1 hour — common waste।
    • 
          
  • Auto-shutdown idle notebooks (Kubeflow Notebooks-এ feature)।
    • 
          
  • "Use it or lose it" policy।
    • 
        

        
Bangladesh context note:

        
    
          
  • GPU expensive ($1.5-3/hour A100)। Sharing critical।
    • 
          
  • Local cloud (BDIX, BD-tier providers) GPU offerings limited।
    • 
          
  • Hybrid common: short jobs cloud, long-batch on-prem।
    • 
        

        

          মূল উপলব্ধি: Multi-team GPU sharing — pure technical না, organizational governance-ও। Quota + priority + chargeback combined — fair pattern। MIG + autoscaling + idle shutdown — efficiency। Single tool magic না — combination।
        

      

    

  


  

    
 অনুশীলন

    
    
      
  1. 
        Manifest: উপরের sentiment-api manifest নিয়ে — replicas বাড়িয়ে ৫ করুন, একটি ConfigMap যোগ করুন (model URL store), ও env-এ ব্যবহার করুন।
        
    
          
          
    
            
    apiVersion: v1
kind: ConfigMap
metadata: { name: sentiment-config }
data:
  MODEL_URL: "s3://my-bucket/models/sentiment-v1.3.0.joblib"
---
# in container env:
envFrom:
  - configMapRef: { name: sentiment-config }
    
          
    
        
    
      
    2. 
      
  2. 
        চিন্তা: কখন আপনি একটি ML team-কে K8s-এ যেতে পরামর্শ দেবেন না? ৩টি specific scenario।
        
    
          
          
    
            
      
              
    • একটি model, monthly batch — Cloud Function/Lambda যথেষ্ট।
      • 
              
    • Team-এ DevOps expertise নেই + dedicated infra-এর budget নেই।
      • 
              
    • Ultra-low-latency edge — K8s overhead ms matters।
      • 
            
    
          
    
        
    
      
    3. 
      
  3. 
        Diagnose: একটি ML pod CrashLoopBackOff state-এ। কী ৩টি command চালাবেন debug-এ?
        
    
          
          
    
            
      
              
    • kubectl describe pod <name> — events, scheduling problem।
      • 
              
    • kubectl logs <name> --previous — previous crash log।
      • 
              
    • kubectl exec -it <name> -- /bin/bash — interactive (যদি pod চলতে থাকে)। OR check resource limits/probes।
      • 
            
    
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন

    
  







  
  
     এই পাঠে ভুল পেয়েছেন? জানান
  





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ৬ · Docker for ML