পাঠ ১৯ · ৩৩-এর মধ্যে · মডিউল ৩

Home / AI Courses / MLOps / Batch vs online

Batch বনাম Online inference

Batch vs online — when to use which

৬ মিনিট পড়া মধ্য · Intermediate Concept

এই পাঠে যা শিখবেন

Batch ও online inference-এর pros/cons
Cost comparison framework
"Near-real-time" middle ground patterns
Bangladesh use case mapping

১ · Two ends of spectrum

Batch ও online — same model, vastly different infrastructure.

Batch example: Daraz রাতে সব user-এর recommendation pre-compute. Spark job. Result Postgres-এ stored. Web request → DB lookup, no model call.
Online example: bKash transaction → real-time fraud check. FastAPI → model → response 50ms-এ।

২ · Trade-off matrix

Latency: batch hours; online ms.
Cost per prediction: batch much cheaper (no idle, parallel).
Always-on infra cost: batch zero between runs; online 24×7.
Freshness: batch stale up to refresh interval; online instant.
Operational: batch simpler (no SLA stress); online complex (autoscale, monitoring).
Failure recovery: batch rerun OK; online incidents user-facing.

৩ · Cost comparison

$$ \text{Cost}_{\text{online}} = \text{server\_cost\_per\_hour} \times 24 \times 30 \times \text{replicas} $$

$$ \text{Cost}_{\text{batch}} = \text{compute\_per\_run} \times \text{runs\_per\_month} $$

Example: 1M users predict monthly.

Online: 3 pods × $50/month each = $150. Always-on.
Batch: Spark cluster ৩ ঘণ্টা × $5/hour × 30 runs (daily) = $450. Or weekly = $60.
For 1M predictions/month — batch weekly cheapest, but freshness 7 days.

৪ · "Near-real-time" — middle ground

Many use case-এ "exact real-time" overkill, "daily batch" too stale.

Mini-batch every 5 minutes: Spark Streaming, Flink.
Pre-compute + override: daily batch baseline, real-time signal overlay.
Lambda architecture: batch + speed layer combined.

৫ · When batch is right

"All users at once" — recommendation, segment, score.
Latency relaxed (24h-7d staleness OK).
Predictable volume — easy to size Spark.
Cost-conscious, GPU expensive online.
Result consumed asynchronously (email, dashboard).

৬ · When online is right

Per-request, user-context-dependent.
Freshness critical — real-time signal matter.
Low SLA latency (<100ms).
Personalization heavy.
Fraud/security — instant decision needed.

৭ · Hybrid common

Production reality often blend.

Recommendation: batch nightly + online re-rank.
Search: pre-computed embeddings + real-time scoring.
Fraud: online check + batch periodic deep audit.

৮ · Batch implementation example

Python · Batch inference

import pandas as pd
from sqlalchemy import create_engine
import joblib

# 1. Load model
model = joblib.load("models/recommendation-v2.joblib")

# 2. Load all users (chunked for memory)
engine = create_engine("postgresql://...")
chunk_size = 100_000

predictions_total = []
for chunk in pd.read_sql_query(
    "SELECT user_id, * FROM user_features", engine, chunksize=chunk_size
):
    # 3. Predict in batch (much faster than per-row)
    scores = model.predict_proba(chunk.drop(columns=["user_id"]))[:, 1]
    chunk_pred = pd.DataFrame({
        "user_id": chunk["user_id"],
        "score": scores,
        "model_version": "v2.0",
        "prediction_date": pd.Timestamp.now(),
    })
    predictions_total.append(chunk_pred)

# 4. Persist
result = pd.concat(predictions_total)
result.to_sql("user_predictions", engine, if_exists="replace", index=False)
print(f"✓ {len(result)} predictions stored")

Pattern: chunked read for large dataset, batch predict (vectorized), bulk write. Spark version-এ much higher scale possible.

Batch — pre-compute, cache; Online — per-request real-time. Production-এ অনেক time hybrid.

Bangladesh-এ small startup-এ batch dominant (cost). Mid-large fintech-এ online dominant (latency-critical). বুদ্ধিমান team workload-anuযায়ী mix.

ভাবনার প্রশ্ন

প্র ০১"Cost calculator — concrete BD scenario."

Bangladesh e-commerce — 5M users, daily recommendation.

Online scenario:

Average ৫০ recommendation requests/user/day = ২৫০M predictions/day.
p99 latency ১০০ms; ৩০ pods × ১,০০০ RPS = ৩০,০০০ RPS capacity.
Pod cost: $৫০/month × ৩০ = $১,৫০০/month = ~১.৭ লাখ BDT.

Batch scenario:

5M users × 100 candidate items each = 500M predictions/day.
Spark cluster: ১০ nodes × ৩ hours × $২/hour = $৬০/day = $১,৮০০/month.
Result store: Postgres/Redis ~$১০০/month.
Total: ~$১,৯০০ = ~২.২ লাখ BDT.

Hybrid (most realistic):

Batch nightly (cheap baseline).
Online re-rank top-10 with real-time context (small online cost).
Total: ~$১,২০০/month — best of both.

Insights:

Online cheaper at LOW request volume (idle waste matters).
Batch wins at HIGH volume (parallel efficiency).
Hybrid usually wins comprehensively.

মূল উপলব্ধি: Cost depends not just compute, but request profile. Always-on infra-এর idle cost batch-এ আসে না। Hybrid often dominates pure choice.

প্র ০২"Cold cache — batch result yet পৌঁছায়নি; কীভাবে handle?"

Batch architecture-এ classic problem.

Scenario:

New user signed up at 11 AM.
Batch ran last night 2 AM.
Recommendation cache empty for this user.

Handling strategies:

Default fallback: popular items, all-user trending.
Cohort-based fallback: "users like you" — based on signup attributes.
Online fallback: miss cache → online inference real-time.
Async backfill: trigger batch for new user immediately.

Production pattern:

Tier 1: cached batch result.
Tier 2: online inference for cache miss.
Tier 3: hard-coded fallback (most popular).

BD example — Daraz signup:

0-1 min after signup: top-10 platform popular.
1-30 min: cohort-based (similar profiles).
After first batch refresh (within 24h): personalized.

Monitoring:

Cache miss rate metric.
Fallback usage trend — sudden spike = batch issue.

মূল উপলব্ধি: Batch staleness expected; gracefully degrade. Multi-tier fallback essential. User never sees blank.

প্র ০৩"Batch warming online baseline — useful?"

Hybrid pattern — batch result-কে online cache hot-load করা।

Pattern:

Batch nightly compute baseline scores.
Online service startup-এ Redis-এ load.
Real-time event update Redis incrementally.
API: real-time-ifs-fresh, fallback-batch.

Benefits:

Cold start fast (Redis pre-loaded).
Real-time updates during day.
Best baseline + best freshness.

Complexity:

Two pipelines maintain.
Cache invalidation careful.
Schema match across both.

BD example — Pathao surge:

Batch hourly: zone-level demand baseline.
Streaming: real-time ride event update.
Inference time: combine both.

মূল উপলব্ধি: Lambda architecture practical for ML. Batch foundation + streaming live + serving combine = production-grade.

প্র ০৪"Migration — batch থেকে online; কখন trigger?"

Pattern migration triggered by clear signal.

Batch → Online signals:

User feedback — "stale recommendations".
A/B test — fresher data variant wins.
Real-time signal underutilized — recent click impacts conversion massively.
Per-context personalization need (different result for same user, different time).

Online → Batch signals:

Cost analysis — idle cost too high.
Latency relaxed — staleness acceptable for some use cases.
Volume burst — batch better at peak.

Migration strategy:

Don't all-at-once. Hybrid period.
New use case — try better pattern first.
Existing well-running — don't touch.

Cost of migration:

Different infrastructure.
Different observability.
Team retraining.
Data flow rework.
Rule of thumb: 2-4 engineer-month for non-trivial migration.

মূল উপলব্ধি: Migration triggered by clear cost/latency signal — not preference. Yearly architecture review healthy practice.

অনুশীলন

Calculate: আপনার (real or imagined) ML use case-এর batch vs online cost.
Volume × cost-per-prediction + always-on infra → comparison. Hybrid often dominant.
Design: Foodpanda-র "delivery time prediction" — batch, online, hybrid?
Hybrid: batch driver/restaurant base profiles; online real-time traffic + weather signal. Combine at request time.
চিন্তা: 5-tier fallback strategy bKash fraud-এর জন্য।
- Tier 1: real-time online model.
- Tier 2: cached recent prediction (minutes old).
- Tier 3: rule-based override.
- Tier 4: amount-based threshold.
- Tier 5: human review queue.