পাঠ ১৮ · ৩৩-এর মধ্যে · মডিউল ৩

Home / AI Courses / MLOps / BentoML & TorchServe

BentoML ও TorchServe

BentoML & TorchServe — alternative serving frameworks

৭ মিনিট পড়া মধ্য · Intermediate Python

এই পাঠে যা শিখবেন

BentoML Service + Runner pattern
TorchServe handler architecture
FastAPI/Triton/BentoML/TorchServe trade-offs
Choose criteria for Bangladesh teams

১ · BentoML overview

BentoML — Python-native ML serving framework. FastAPI-এর বদলে dedicated ML serving abstraction.

Service decorator + Runner abstraction.
IO descriptors — input/output schema.
bentoml build — packages model + code + dependencies.
Yatai = BentoML's K8s deployment platform.

২ · BentoML Service example

Python · BentoML service

import bentoml
from bentoml.io import JSON, NumpyNdarray
import numpy as np
from pydantic import BaseModel

# 1. Save model to BentoML store
# bentoml.sklearn.save_model("bangla_sentiment", trained_model)

# 2. Create runner
sentiment_runner = bentoml.sklearn.get("bangla_sentiment:latest").to_runner()

svc = bentoml.Service(
    "bangla_sentiment_service",
    runners=[sentiment_runner],
)

class ReviewIn(BaseModel):
    text: str

@svc.api(input=JSON(pydantic_model=ReviewIn), output=JSON())
async def predict(review: ReviewIn) -> dict:
    proba = await sentiment_runner.predict_proba.async_run([review.text])
    label_idx = proba[0].argmax()
    return {
        "label": str(label_idx),
        "confidence": float(proba[0][label_idx]),
    }

# Run:
# bentoml serve service:svc
# Build:
# bentoml build
# Containerize:
# bentoml containerize bangla_sentiment_service:latest

Service handles HTTP layer; Runner = isolated worker process running model. Async flow built-in. Single command bento build → Docker image.

৩ · BentoML benefits

Framework agnostic: sklearn, PyTorch, TF, XGBoost, ONNX — all supported uniformly.
Adaptive batching: dynamic batching with auto-tuning.
Built-in observability: Prometheus metrics, Grafana dashboards.
bento format: code + model + config + dependencies — single deployable artifact.
Yatai: K8s deployment, model registry, Bento store.

৪ · TorchServe overview

TorchServe — PyTorch official model serving. AWS-Meta collaboration (originally).

Handler-based: BaseHandler subclass — preprocess, inference, postprocess methods.
Model Archiver (.mar): model + handler + extra files.
Multi-worker via configuration.

৫ · TorchServe handler example

Python · TorchServe handler

from ts.torch_handler.base_handler import BaseHandler
import torch
from transformers import AutoTokenizer

class BanglaSentimentHandler(BaseHandler):
    def initialize(self, context):
        super().initialize(context)
        self.tokenizer = AutoTokenizer.from_pretrained("bangla-bert-base")

    def preprocess(self, requests):
        texts = [req.get("body").get("text") for req in requests]
        return self.tokenizer(
            texts, padding=True, truncation=True,
            max_length=128, return_tensors="pt"
        ).to(self.device)

    def inference(self, model_input):
        with torch.no_grad():
            outputs = self.model(**model_input)
        return outputs.logits

    def postprocess(self, inference_output):
        probs = torch.softmax(inference_output, dim=-1)
        labels = probs.argmax(dim=-1).tolist()
        confs = probs.max(dim=-1).values.tolist()
        return [
            {"label": str(l), "confidence": c}
            for l, c in zip(labels, confs)
        ]

# Package:
# torch-model-archiver --model-name bangla --version 1.0 \
#   --serialized-file model.pt --handler bangla_handler.py \
#   --extra-files config.json
# Serve:
# torchserve --start --model-store . --models bangla=bangla.mar

Handler design — preprocess, inference, postprocess separated. Multi-worker autoscaling. Built-in metrics endpoint.

৬ · Comparison table

FastAPI: general Python web; ML-aware via custom. Most flexible, most boilerplate.
Triton: GPU-optimized, multi-framework, dynamic batching. Best for high-RPS GPU.
BentoML: Python-native, framework-agnostic, decent DX. Good middle ground.
TorchServe: PyTorch-deeply-integrated. Official tool, conservative choice.
KServe: K8s-native serving (CRD-based). Higher-level than all.

ML serving frameworks — flexibility থেকে specialization spectrum. Project-এর scale ও workload type অনুযায়ী choose.

Bangladesh adoption: FastAPI dominant (~৭০%), Triton growing in DL teams (~১৫%), BentoML rising (~১০%), TorchServe niche (~৫%).

ভাবনার প্রশ্ন

প্র ০১"BentoML vs FastAPI — when to switch?"

BentoML extra abstraction; worth pivoting?

Stick with FastAPI when:

Single, simple model.
Custom request flow heavy.
Team Python web-comfortable.

Switch to BentoML when:

Multiple models share infrastructure.
"Bento build → docker → deploy" automation valuable.
Adaptive batching needed (BentoML built-in).
K8s deploy frequently (Yatai automate).

Migration cost:

Code rewrite — 1-2 days per service.
Build/CI rework.
Team learn Service + Runner pattern.

BD context:

Most teams stay FastAPI. Switch trigger usually multi-model + DX pain.

মূল উপলব্ধি: BentoML = ML-purpose-built; FastAPI = general. Pivot cost-justified at scale; default FastAPI fine for most.

প্র ০২"TorchServe still relevant in 2025?"

TorchServe — was hot, now seems stagnant.

Still relevant when:

Pure PyTorch shop, conservative tooling preference.
AWS Sagemaker workflow (deeply integrated).
Existing TorchServe stack — don't break what works.

Less relevant due to:

Triton overlaps + does more (multi-framework, GPU).
BentoML easier dev UX.
FastAPI + custom code flexible.
Slower release cadence than alternatives.

Choose new project when?

Rarely first choice today.
SageMaker bring TorchServe automatically — that's main use case.

Future:

vLLM + TGI (text generation server) for LLM-specific overshadow.
TorchServe relegated to legacy PyTorch CV-style.

মূল উপলব্ধি: TorchServe — viable but not default 2025. Specialized SageMaker + PyTorch. New projects: Triton, BentoML, FastAPI generally better.

প্র ০৩"vLLM ও TGI — LLM-specific serving কেন?"

LLM serving traditional frameworks-এর gap revealed.

LLM serving challenges:

Variable output length — fixed batching inefficient.
KV cache memory dominant.
Streaming output — token-by-token return needed.
Quantization (GPTQ, AWQ) integration.

vLLM (UC Berkeley):

PagedAttention — KV cache memory management innovation.
Continuous batching — variable-length output efficient.
OpenAI-compatible API.
2-10× throughput vs naive PyTorch.

Hugging Face TGI (Text Generation Inference):

HF-native LLM server.
Built-in quantization, flash attention.
Streaming, batching.

BD relevance:

Bangla LLM serving (BanglaT5, BanglaGPT) — vLLM appropriate.
Self-hosted LLaMA/Mistral fine-tunes — vLLM standard.
GPU cost-conscious — throughput matters.

Triton vs vLLM:

Triton supports vLLM as backend (TRT-LLM).
Standalone vLLM simpler if LLM-only use case.

মূল উপলব্ধি: LLM serving = sub-domain own tooling. vLLM/TGI standard for self-hosted LLM 2025. Triton for heterogeneous; vLLM for LLM-focused.

প্র ০৪"KServe — K8s CRD layer add value?"

KServe (formerly KFServing) — K8s-native ML serving CRD layer.

What it does:

Define InferenceService CRD.
Behind: choose runtime (Triton, TorchServe, sklearn-server, custom).
K8s autoscale (Knative-based zero-scale).
Built-in canary, multi-model.

Pros:

Declarative — minimal YAML, model URL, request CPU/GPU.
Multi-runtime — same K8s manifest, different backend.
Scale-to-zero — idle cost zero.

Cons:

Knative dependency — additional K8s complexity.
Cold start (zero-scale) painful for ML.
Abstraction cost — debug deeper sometimes.

Choose KServe when:

Many models, frequent provisioning.
K8s expertise strong.
Multi-team self-service desired.
Cost optimization (zero-scale) matters.