পাঠ ১৮ · ৩৩-এর মধ্যে · মডিউল ৩

Home / AI Courses / MLOps / BentoML & TorchServe

BentoML ও TorchServe

BentoML & TorchServe — alternative serving frameworks

৭ মিনিট পড়া মধ্য · Intermediate Python

এই পাঠে যা শিখবেন

BentoML Service + Runner pattern
TorchServe handler architecture
FastAPI/Triton/BentoML/TorchServe trade-offs
Choose criteria for Bangladesh teams

১ · BentoML overview

BentoML — Python-native ML serving framework। FastAPI-এর বদলে dedicated ML serving abstraction।

Service decorator + Runner abstraction।
IO descriptors — input/output schema।
bentoml build — packages model + code + dependencies।
Yatai = BentoML's K8s deployment platform।

২ · BentoML Service example

Python · BentoML service

import bentoml
from bentoml.io import JSON, NumpyNdarray
import numpy as np
from pydantic import BaseModel

# 1. Save model to BentoML store
# bentoml.sklearn.save_model("bangla_sentiment", trained_model)

# 2. Create runner
sentiment_runner = bentoml.sklearn.get("bangla_sentiment:latest").to_runner()

svc = bentoml.Service(
    "bangla_sentiment_service",
    runners=[sentiment_runner],
)

class ReviewIn(BaseModel):
    text: str

@svc.api(input=JSON(pydantic_model=ReviewIn), output=JSON())
async def predict(review: ReviewIn) -> dict:
    proba = await sentiment_runner.predict_proba.async_run([review.text])
    label_idx = proba[0].argmax()
    return {
        "label": str(label_idx),
        "confidence": float(proba[0][label_idx]),
    }

# Run:
# bentoml serve service:svc
# Build:
# bentoml build
# Containerize:
# bentoml containerize bangla_sentiment_service:latest

Service handles HTTP layer; Runner = isolated worker process running model। Async flow built-in। Single command bento build → Docker image।

৩ · BentoML benefits

Framework agnostic: sklearn, PyTorch, TF, XGBoost, ONNX — all supported uniformly।
Adaptive batching: dynamic batching with auto-tuning।
Built-in observability: Prometheus metrics, Grafana dashboards।
bento format: code + model + config + dependencies — single deployable artifact।
Yatai: K8s deployment, model registry, Bento store।

৪ · TorchServe overview

TorchServe — PyTorch official model serving। AWS-Meta collaboration (originally)।

Handler-based: BaseHandler subclass — preprocess, inference, postprocess methods।
Model Archiver (.mar): model + handler + extra files।
Multi-worker via configuration।

৫ · TorchServe handler example

Python · TorchServe handler

from ts.torch_handler.base_handler import BaseHandler
import torch
from transformers import AutoTokenizer

class BanglaSentimentHandler(BaseHandler):
    def initialize(self, context):
        super().initialize(context)
        self.tokenizer = AutoTokenizer.from_pretrained("bangla-bert-base")

    def preprocess(self, requests):
        texts = [req.get("body").get("text") for req in requests]
        return self.tokenizer(
            texts, padding=True, truncation=True,
            max_length=128, return_tensors="pt"
        ).to(self.device)

    def inference(self, model_input):
        with torch.no_grad():
            outputs = self.model(**model_input)
        return outputs.logits

    def postprocess(self, inference_output):
        probs = torch.softmax(inference_output, dim=-1)
        labels = probs.argmax(dim=-1).tolist()
        confs = probs.max(dim=-1).values.tolist()
        return [
            {"label": str(l), "confidence": c}
            for l, c in zip(labels, confs)
        ]

# Package:
# torch-model-archiver --model-name bangla --version 1.0 \
#   --serialized-file model.pt --handler bangla_handler.py \
#   --extra-files config.json
# Serve:
# torchserve --start --model-store . --models bangla=bangla.mar

Handler design — preprocess, inference, postprocess separated। Multi-worker autoscaling। Built-in metrics endpoint।

৬ · Comparison table

FastAPI: general Python web; ML-aware via custom। Most flexible, most boilerplate।
Triton: GPU-optimized, multi-framework, dynamic batching। Best for high-RPS GPU।
BentoML: Python-native, framework-agnostic, decent DX। Good middle ground।
TorchServe: PyTorch-deeply-integrated। Official tool, conservative choice।
KServe: K8s-native serving (CRD-based). Higher-level than all।

ML serving frameworks — flexibility থেকে specialization spectrum। Project-এর scale ও workload type অনুযায়ী choose।

Bangladesh adoption: FastAPI dominant (~৭০%), Triton growing in DL teams (~১৫%), BentoML rising (~১০%), TorchServe niche (~৫%)।

ভাবনার প্রশ্ন

প্র ০১"BentoML vs FastAPI — when to switch?"

BentoML extra abstraction; worth pivoting?

Stick with FastAPI when:

Single, simple model।
Custom request flow heavy।
Team Python web-comfortable।

Switch to BentoML when:

Multiple models share infrastructure।
"Bento build → docker → deploy" automation valuable।
Adaptive batching needed (BentoML built-in)।
K8s deploy frequently (Yatai automate)।

Migration cost:

Code rewrite — 1-2 days per service।
Build/CI rework।
Team learn Service + Runner pattern।

BD context:

Most teams stay FastAPI। Switch trigger usually multi-model + DX pain।

মূল উপলব্ধি: BentoML = ML-purpose-built; FastAPI = general। Pivot cost-justified at scale; default FastAPI fine for most।

প্র ০২"TorchServe still relevant in 2025?"

TorchServe — was hot, now seems stagnant।

Still relevant when:

Pure PyTorch shop, conservative tooling preference।
AWS Sagemaker workflow (deeply integrated)।
Existing TorchServe stack — don't break what works।

Less relevant due to:

Triton overlaps + does more (multi-framework, GPU)।
BentoML easier dev UX।
FastAPI + custom code flexible।
Slower release cadence than alternatives।

Choose new project when?

Rarely first choice today।
SageMaker bring TorchServe automatically — that's main use case।

Future:

vLLM + TGI (text generation server) for LLM-specific overshadow।
TorchServe relegated to legacy PyTorch CV-style।

মূল উপলব্ধি: TorchServe — viable but not default 2025। Specialized SageMaker + PyTorch. New projects: Triton, BentoML, FastAPI generally better।

প্র ০৩"vLLM ও TGI — LLM-specific serving কেন?"

LLM serving traditional frameworks-এর gap revealed।

LLM serving challenges:

Variable output length — fixed batching inefficient।
KV cache memory dominant।
Streaming output — token-by-token return needed।
Quantization (GPTQ, AWQ) integration।

vLLM (UC Berkeley):

PagedAttention — KV cache memory management innovation।
Continuous batching — variable-length output efficient।
OpenAI-compatible API।
2-10× throughput vs naive PyTorch।

Hugging Face TGI (Text Generation Inference):

HF-native LLM server।
Built-in quantization, flash attention।
Streaming, batching।

BD relevance:

Bangla LLM serving (BanglaT5, BanglaGPT) — vLLM appropriate।
Self-hosted LLaMA/Mistral fine-tunes — vLLM standard।
GPU cost-conscious — throughput matters।

Triton vs vLLM:

Triton supports vLLM as backend (TRT-LLM)।
Standalone vLLM simpler if LLM-only use case।

মূল উপলব্ধি: LLM serving = sub-domain own tooling। vLLM/TGI standard for self-hosted LLM 2025। Triton for heterogeneous; vLLM for LLM-focused।

প্র ০৪"KServe — K8s CRD layer add value?"

KServe (formerly KFServing) — K8s-native ML serving CRD layer।

What it does:

Define InferenceService CRD।
Behind: choose runtime (Triton, TorchServe, sklearn-server, custom)।
K8s autoscale (Knative-based zero-scale)।
Built-in canary, multi-model।

Pros:

Declarative — minimal YAML, model URL, request CPU/GPU।
Multi-runtime — same K8s manifest, different backend।
Scale-to-zero — idle cost zero।

Cons:

Knative dependency — additional K8s complexity।
Cold start (zero-scale) painful for ML।
Abstraction cost — debug deeper sometimes।

Choose KServe when:

Many models, frequent provisioning।
K8s expertise strong।
Multi-team self-service desired।
Cost optimization (zero-scale) matters।