পাঠ ১৯ · ৩৩-এর মধ্যে · মডিউল ৩
Home / AI Courses / MLOps / Batch vs online

Batch বনাম Online inference

Batch vs online — when to use which
৬ মিনিট পড়া মধ্য · Intermediate Concept

এই পাঠে যা শিখবেন

  • Batch ও online inference-এর pros/cons
  • Cost comparison framework
  • "Near-real-time" middle ground patterns
  • Bangladesh use case mapping

১ · Two ends of spectrum

Batch ও online — same model, vastly different infrastructure।

  • Batch example: Daraz রাতে সব user-এর recommendation pre-compute। Spark job। Result Postgres-এ stored। Web request → DB lookup, no model call।
  • Online example: bKash transaction → real-time fraud check। FastAPI → model → response 50ms-এ।

২ · Trade-off matrix

  • Latency: batch hours; online ms।
  • Cost per prediction: batch much cheaper (no idle, parallel)।
  • Always-on infra cost: batch zero between runs; online 24×7।
  • Freshness: batch stale up to refresh interval; online instant।
  • Operational: batch simpler (no SLA stress); online complex (autoscale, monitoring)।
  • Failure recovery: batch rerun OK; online incidents user-facing।

৩ · Cost comparison

$$ \text{Cost}_{\text{online}} = \text{server\_cost\_per\_hour} \times 24 \times 30 \times \text{replicas} $$

$$ \text{Cost}_{\text{batch}} = \text{compute\_per\_run} \times \text{runs\_per\_month} $$

Example: 1M users predict monthly।

  • Online: 3 pods × $50/month each = $150। Always-on।
  • Batch: Spark cluster ৩ ঘণ্টা × $5/hour × 30 runs (daily) = $450। Or weekly = $60।
  • For 1M predictions/month — batch weekly cheapest, but freshness 7 days।

৪ · "Near-real-time" — middle ground

Many use case-এ "exact real-time" overkill, "daily batch" too stale।

  • Mini-batch every 5 minutes: Spark Streaming, Flink।
  • Pre-compute + override: daily batch baseline, real-time signal overlay।
  • Lambda architecture: batch + speed layer combined।

৫ · When batch is right

  • "All users at once" — recommendation, segment, score।
  • Latency relaxed (24h-7d staleness OK)।
  • Predictable volume — easy to size Spark।
  • Cost-conscious, GPU expensive online।
  • Result consumed asynchronously (email, dashboard)।

৬ · When online is right

  • Per-request, user-context-dependent।
  • Freshness critical — real-time signal matter।
  • Low SLA latency (<100ms)।
  • Personalization heavy।
  • Fraud/security — instant decision needed।

৭ · Hybrid common

Production reality often blend।

  • Recommendation: batch nightly + online re-rank।
  • Search: pre-computed embeddings + real-time scoring।
  • Fraud: online check + batch periodic deep audit।

৮ · Batch implementation example

Python · Batch inference 
import pandas as pd
from sqlalchemy import create_engine
import joblib

# 1. Load model
model = joblib.load("models/recommendation-v2.joblib")

# 2. Load all users (chunked for memory)
engine = create_engine("postgresql://...")
chunk_size = 100_000

predictions_total = []
for chunk in pd.read_sql_query(
    "SELECT user_id, * FROM user_features", engine, chunksize=chunk_size
):
    # 3. Predict in batch (much faster than per-row)
    scores = model.predict_proba(chunk.drop(columns=["user_id"]))[:, 1]
    chunk_pred = pd.DataFrame({
        "user_id": chunk["user_id"],
        "score": scores,
        "model_version": "v2.0",
        "prediction_date": pd.Timestamp.now(),
    })
    predictions_total.append(chunk_pred)

# 4. Persist
result = pd.concat(predictions_total)
result.to_sql("user_predictions", engine, if_exists="replace", index=False)
print(f"✓ {len(result)} predictions stored")

    


    

      Pattern: chunked read for large dataset, batch predict (vectorized), bulk write। Spark version-এ much higher scale possible।
    


    

      
        
        Batch vs Online inference architecture

        
        Batch (nightly)
        
        Data lake
        all users

        
        Spark job
        3 hours

        
        DB cache
        user → score

        
        

        Web request → DB lookup, no model call

        
        Online (real-time)

        
        User request

        
        FastAPI/Triton
        50ms

        
        

        Per-request, always-on infra

        
        
        Batch
        Online
        

        Latency:
        hours
        milliseconds

        Cost/M pred:
        $0.5-2
        $10-50

        Freshness:
        stale (24h)
        instant

        Use case:
        recommendation, segment
        fraud, search ranking
      
      
Batch — pre-compute, cache; Online — per-request real-time। Production-এ অনেক time hybrid।

    


    

      Bangladesh-এ small startup-এ batch dominant (cost)। Mid-large fintech-এ online dominant (latency-critical)। বুদ্ধিমান team workload-anuযায়ী mix।
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      প্র ০১"Cost calculator — concrete BD scenario।"
      

        
Bangladesh e-commerce — 5M users, daily recommendation।

        
Online scenario:

        
    
          
  • Average ৫০ recommendation requests/user/day = ২৫০M predictions/day।
    • 
          
  • p99 latency ১০০ms; ৩০ pods × ১,০০০ RPS = ৩০,০০০ RPS capacity।
    • 
          
  • Pod cost: $৫০/month × ৩০ = $১,৫০০/month = ~১.৭ লাখ BDT।
    • 
        

        
Batch scenario:

        
    
          
  • 5M users × 100 candidate items each = 500M predictions/day।
    • 
          
  • Spark cluster: ১০ nodes × ৩ hours × $২/hour = $৬০/day = $১,৮০০/month।
    • 
          
  • Result store: Postgres/Redis ~$১০০/month।
    • 
          
  • Total: ~$১,৯০০ = ~২.২ লাখ BDT।
    • 
        

        
Hybrid (most realistic):

        
    
          
  • Batch nightly (cheap baseline)।
    • 
          
  • Online re-rank top-10 with real-time context (small online cost)।
    • 
          
  • Total: ~$১,২০০/month — best of both।
    • 
        

        
Insights:

        
    
          
  • Online cheaper at LOW request volume (idle waste matters)।
    • 
          
  • Batch wins at HIGH volume (parallel efficiency)।
    • 
          
  • Hybrid usually wins comprehensively।
    • 
        

        
মূল উপলব্ধি: Cost depends not just compute, but request profile। Always-on infra-এর idle cost batch-এ আসে না। Hybrid often dominates pure choice।

      

    


    

      প্র ০২"Cold cache — batch result yet পৌঁছায়নি; কীভাবে handle?"
      

        
Batch architecture-এ classic problem।

        
Scenario:

        
    
          
  • New user signed up at 11 AM।
    • 
          
  • Batch ran last night 2 AM।
    • 
          
  • Recommendation cache empty for this user।
    • 
        

        
Handling strategies:

        
    
          
  • Default fallback: popular items, all-user trending।
    • 
          
  • Cohort-based fallback: "users like you" — based on signup attributes।
    • 
          
  • Online fallback: miss cache → online inference real-time।
    • 
          
  • Async backfill: trigger batch for new user immediately।
    • 
        

        
Production pattern:

        
    
          
  • Tier 1: cached batch result।
    • 
          
  • Tier 2: online inference for cache miss।
    • 
          
  • Tier 3: hard-coded fallback (most popular)।
    • 
        

        
BD example — Daraz signup:

        
    
          
  • 0-1 min after signup: top-10 platform popular।
    • 
          
  • 1-30 min: cohort-based (similar profiles)।
    • 
          
  • After first batch refresh (within 24h): personalized।
    • 
        

        
Monitoring:

        
    
          
  • Cache miss rate metric।
    • 
          
  • Fallback usage trend — sudden spike = batch issue।
    • 
        

        
মূল উপলব্ধি: Batch staleness expected; gracefully degrade। Multi-tier fallback essential। User never sees blank।

      

    


    

      প্র ০৩"Batch warming online baseline — useful?"
      

        
Hybrid pattern — batch result-কে online cache hot-load করা।

        
Pattern:

        
    
          
  • Batch nightly compute baseline scores।
    • 
          
  • Online service startup-এ Redis-এ load।
    • 
          
  • Real-time event update Redis incrementally।
    • 
          
  • API: real-time-ifs-fresh, fallback-batch।
    • 
        

        
Benefits:

        
    
          
  • Cold start fast (Redis pre-loaded)।
    • 
          
  • Real-time updates during day।
    • 
          
  • Best baseline + best freshness।
    • 
        

        
Complexity:

        
    
          
  • Two pipelines maintain।
    • 
          
  • Cache invalidation careful।
    • 
          
  • Schema match across both।
    • 
        

        
BD example — Pathao surge:

        
    
          
  • Batch hourly: zone-level demand baseline।
    • 
          
  • Streaming: real-time ride event update।
    • 
          
  • Inference time: combine both।
    • 
        

        
মূল উপলব্ধি: Lambda architecture practical for ML। Batch foundation + streaming live + serving combine = production-grade।

      

    


    

      প্র ০৪"Migration — batch থেকে online; কখন trigger?"
      

        
Pattern migration triggered by clear signal।

        
Batch → Online signals:

        
    
          
  • User feedback — "stale recommendations"।
    • 
          
  • A/B test — fresher data variant wins।
    • 
          
  • Real-time signal underutilized — recent click impacts conversion massively।
    • 
          
  • Per-context personalization need (different result for same user, different time)।
    • 
        

        
Online → Batch signals:

        
    
          
  • Cost analysis — idle cost too high।
    • 
          
  • Latency relaxed — staleness acceptable for some use cases।
    • 
          
  • Volume burst — batch better at peak।
    • 
        

        
Migration strategy:

        
    
          
  • Don't all-at-once। Hybrid period।
    • 
          
  • New use case — try better pattern first।
    • 
          
  • Existing well-running — don't touch।
    • 
        

        
Cost of migration:

        
    
          
  • Different infrastructure।
    • 
          
  • Different observability।
    • 
          
  • Team retraining।
    • 
          
  • Data flow rework।
    • 
          
  • Rule of thumb: 2-4 engineer-month for non-trivial migration।
    • 
        

        
মূল উপলব্ধি: Migration triggered by clear cost/latency signal — not preference। Yearly architecture review healthy practice।

      

    

  


  

    
 অনুশীলন

    
    
      
  1. Calculate: আপনার (real or imagined) ML use case-এর batch vs online cost।
        
    Volume × cost-per-prediction + always-on infra → comparison। Hybrid often dominant।
    
      
    2. 
      
  2. Design: Foodpanda-র "delivery time prediction" — batch, online, hybrid?
        
    Hybrid: batch driver/restaurant base profiles; online real-time traffic + weather signal। Combine at request time।
    
      
    3. 
      
  3. চিন্তা: 5-tier fallback strategy bKash fraud-এর জন্য।
        
    
          
      
            
    • Tier 1: real-time online model।
      • 
            
    • Tier 2: cached recent prediction (minutes old)।
      • 
            
    • Tier 3: rule-based override।
      • 
            
    • Tier 4: amount-based threshold।
      • 
            
    • Tier 5: human review queue।
      • 
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন

    
  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ১৮ · BentoML & TorchServe