পাঠ ১৬ · ৩৩-এর মধ্যে · মডিউল ৩
Home / AI Courses / MLOps / FastAPI serving

FastAPI দিয়ে model API

Production-grade model serving with FastAPI
৮ মিনিট পড়া মধ্য · Intermediate Python

এই পাঠে যা শিখবেন

  • FastAPI app structure — Pydantic, dependency, lifespan
  • Async vs sync endpoint — কখন কোনটা
  • Health probe pattern
  • Batching, error handling, instrumentation

১ · কেন FastAPI

Flask দিয়ে ML serve করা যায়, কিন্তু FastAPI ML-এর জন্য better fit. Pydantic schema validation, async support, OpenAPI auto-documentation. Bangladesh ML team-এর majority FastAPI.

২ · Production-grade serving template

Python · FastAPI complete 
from contextlib import asynccontextmanager
from fastapi import FastAPI, HTTPException, Request
from pydantic import BaseModel, Field
import joblib
import logging
import time
import os

logger = logging.getLogger("ml-api")
logging.basicConfig(level=logging.INFO)

MODEL_VERSION = os.getenv("MODEL_VERSION", "v1.3.0")
MODEL_PATH = os.getenv("MODEL_PATH", f"models/sentiment-{MODEL_VERSION}.joblib")

# --- lifespan: load model once at startup ---
@asynccontextmanager
async def lifespan(app: FastAPI):
    logger.info(f"Loading model {MODEL_VERSION}...")
    app.state.model = joblib.load(MODEL_PATH)
    app.state.ready = True
    logger.info("Model loaded.")
    yield
    # shutdown cleanup
    logger.info("Shutting down.")

app = FastAPI(
    title="Bangla Sentiment API",
    version=MODEL_VERSION,
    lifespan=lifespan,
)

# --- schemas ---
class ReviewIn(BaseModel):
    text: str = Field(..., min_length=1, max_length=2000)

class PredictionOut(BaseModel):
    label: str
    confidence: float
    model_version: str

class BatchIn(BaseModel):
    texts: list[str] = Field(..., min_length=1, max_length=64)

# --- health probes ---
@app.get("/healthz")
def liveness():
    return {"status": "alive"}

@app.get("/readyz")
def readiness(request: Request):
    if not getattr(request.app.state, "ready", False):
        raise HTTPException(503, "Model not loaded")
    return {"status": "ready", "model_version": MODEL_VERSION}

# --- prediction ---
@app.post("/predict", response_model=PredictionOut)
def predict(req: ReviewIn, request: Request):
    start = time.perf_counter()
    try:
        proba = request.app.state.model.predict_proba([req.text])[0]
        label_idx = proba.argmax()
        return PredictionOut(
            label=request.app.state.model.classes_[label_idx],
            confidence=float(proba[label_idx]),
            model_version=MODEL_VERSION,
        )
    except Exception as e:
        logger.exception("predict failed")
        raise HTTPException(500, f"prediction error: {e}")
    finally:
        elapsed_ms = (time.perf_counter() - start) * 1000
        logger.info(f"predict ok in {elapsed_ms:.1f}ms")

# --- batch endpoint ---
@app.post("/predict-batch")
def predict_batch(req: BatchIn, request: Request):
    proba = request.app.state.model.predict_proba(req.texts)
    return {
        "predictions": [
            {"label": request.app.state.model.classes_[p.argmax()], "confidence": float(p.max())}
            for p in proba
        ],
        "model_version": MODEL_VERSION,
    }

    


    

      Pattern features: lifespan model load (load-once), Pydantic schema validation, separate liveness/readiness probes, structured logging, error handling with HTTPException, batch endpoint.
    


    
৩ · Async vs sync


    
    
      
  • Sync (def): CPU-bound model inference (PyTorch, sklearn) — actually fine, FastAPI runs in threadpool.
    • 
      
  • Async (async def): I/O-bound (DB lookup, external API call). Don't make CPU-bound async.
    • 
      
  • Pitfall: async def-এ heavy CPU work → blocks event loop → other request slow.
    • 
      
  • Recommendation: ML inference sync def; pre/post processing if I/O — separate async function.
    • 
    


    
৪ · Health probes detail


    
    
      
  • Liveness (/healthz): "process alive?" — restart trigger যদি fail.
    • 
      
  • Readiness (/readyz): "ready to serve?" — load balancer route trigger.
    • 
      
  • Model load 2-3 মিনিট লাগে — readiness false during, true after.
    • 
      
  • K8s-এ readinessProbe.initialDelaySeconds: 30 realistic.
    • 
    


    
৫ · Workers ও deployment


    

      

        
        Bash · Production launch
      

      
# dev — single process
$ uvicorn main:app --host 0.0.0.0 --port 8000 --reload

# prod — multiple workers (CPU-bound, GIL release per worker)
$ gunicorn main:app -w 4 -k uvicorn.workers.UvicornWorker \
    --bind 0.0.0.0:8000 \
    --timeout 60 \
    --access-logfile -

# K8s — workers per pod, replicas auto-scale
# resources.requests.cpu × workers ≈ pod CPU

      

        
        
      

      

    


    

      Workers/replicas trade-off: bigger model + more memory → fewer workers per pod, more replicas. Model 2 GB-এ 4 workers per pod = 8 GB RAM.
    


    
৬ · Batching


    
Single inference per request CPU-inefficient. Batching idea:

    
    
      
  • Static batch: client sends batch (above /predict-batch).
    • 
      
  • Dynamic batch: server collects requests within window (e.g., 5ms), processes together. Triton/BentoML built-in.
    • 
      
  • Trade-off: batching latency vs throughput.
    • 
    


    
৭ · Common pitfalls


    

      
    
        
  • Model loaded per-request → 100x slower; lifespan-এ load.
    • 
        
  • HTTPException-এ stack trace leak — production-এ generic error message.
    • 
        
  • Logging blocking — async logger / aiologger consider.
    • 
        
  • No timeout — slow request worker tie-up. gunicorn timeout setup.
    • 
      

    


    

      
        
        FastAPI production deployment stack
        Client → Ingress → Pod (Gunicorn → Workers)

        
        Client
        app / browser

        
        Ingress / LB
        readinessProbe

        
        Pod (× N replicas)

        
        Gunicorn master

        
        Worker 1
        
        Worker 2
        
        Worker 3

        
        Each worker — model loaded once (lifespan)

        
        /healthz · /readyz · /predict · /metrics
        probes + business + observability

        
        
      
      
FastAPI production stack — gunicorn manages N workers; each worker model load lifespan-এ; ingress readiness probe-এ traffic route.

    


    

      Bangladesh production-এ এই stack-ই dominant. Triton ব্যবহার GPU-heavy multi-model setup-এ; FastAPI single-model থেকে সরল।
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      প্র ০১"Sync vs async — PyTorch model inference-এ কোনটা?"
      

        
FastAPI-তে common confusion.

        
Sync (def):

        
    
          
  • FastAPI threadpool-এ run; event loop free থাকে।
    • 
          
  • CPU-bound (PyTorch, sklearn) — সঠিক choice.
    • 
          
  • Other I/O-bound endpoints concurrently serve.
    • 
        

        
Async (async def):

        
    
          
  • Event loop-এ run; কোনো heavy CPU work block করে।
    • 
          
  • I/O-bound (DB query, HTTP call) — appropriate.
    • 
          
  • Pitfall: async def + PyTorch — event loop block হবে।
    • 
        

        
Hybrid:

        
    
          
  • Async endpoint যা DB/cache lookup করে — async def.
    • 
          
  • Inference part: await asyncio.to_thread(model.predict, ...) বা run_in_executor.
    • 
        

        
Bangladesh team common mistake:

        
    
          
  • "Async = fast" ভেবে inference async def-এ — actually slower.
    • 
          
  • Sync def + threadpool — same throughput, less complexity.
    • 
        

        
মূল উপলব্ধি: CPU-bound = sync; I/O-bound = async. Mix carefully. Default sync — simpler + correct.

      

    


    

      প্র ০২"Workers কত set করবেন? GIL impact কী PyTorch-এ?"
      

        
Workers tuning crucial production performance.

        
Theoretical: workers = (2 × CPU cores) + 1 (Gunicorn doc).

        
ML reality — GIL:

        
    
          
  • Python GIL — single thread Python at a time.
    • 
          
  • PyTorch heavy ops (matmul) — release GIL during C extension.
    • 
          
  • So PyTorch threading internal benefits (e.g., 4 cores per inference).
    • 
          
  • Multiple workers + each multi-thread → contention.
    • 
        

        
Optimal workers:

        
    
          
  • CPU model (sklearn, lightgbm): workers = cores. Each does one inference at a time.
    • 
          
  • PyTorch CPU: workers = cores / 2 (because PyTorch uses 2-4 threads per inference).
    • 
          
  • GPU: workers = 1-2 per GPU (GPU is bottleneck, not CPU).
    • 
        

        
Memory consideration:

        
    
          
  • Each worker = full model load = full memory.
    • 
          
  • Model 2 GB × 4 workers = 8 GB RAM.
    • 
          
  • Pod resources request match.
    • 
        

        
Tuning approach:

        
    
          
  • Load test (locust, k6) — RPS vs latency curve.
    • 
          
  • Sweet spot: throughput plateau before latency degrades.
    • 
        

        
Common config (mid-size BD service):

        
    
          
  • Pod: 4 CPU, 8 GB RAM, model 1 GB.
    • 
          
  • Workers: 2 (PyTorch) or 4 (sklearn).
    • 
          
  • Replicas: HPA based on RPS.
    • 
        

        
মূল উপলব্ধি: Workers tuning model framework + memory + load profile-এর function. Default formula starting point only — load test-এ তো final say.

      

    


    

      প্র ০৩"Rate limiting + circuit breaker — কোথায় add করবেন?"
      

        
FastAPI-এ resilience patterns.

        
Rate limiting:

        
    
          
  • Per-client RPS limit.
    • 
          
  • Implementations: slowapi (FastAPI-friendly), API gateway (Kong, Tyk).
    • 
          
  • Better at gateway — pod-level redundant.
    • 
        

        
Circuit breaker:

        
    
          
  • External dependency failure detect → fail fast.
    • 
          
  • e.g., Feature Store down — don't hang ৩০s; fast 503.
    • 
          
  • pybreaker, tenacity for retries + circuit.
    • 
        

        
Timeout:

        
    
          
  • External calls: httpx.AsyncClient(timeout=2.0).
    • 
          
  • DB: connection pool timeout.
    • 
          
  • Total request: gunicorn --timeout 60.
    • 
        

        
Bulkhead:

        
    
          
  • Different connection pools per dependency — one slow doesn't block all.
    • 
        

        
Graceful degradation:

        
    
          
  • Feature Store down → use defaults (cached aggregate).
    • 
          
  • Model fallback — simple rule if ML model unavailable.
    • 
        

        
BD context — payment fraud:

        
    
          
  • External enrichment API down — model use without it (less accurate, but transactional flow continues).
    • 
          
  • Vs hard-fail — entire payment refused — user experience disaster.
    • 
        

        
মূল উপলব্ধি: Resilience layered: gateway (rate limit) + service (circuit) + degradation (fallback). Each layer fast-fail, never silent block.

      

    


    

      প্র ০৪"Logging — request body log করা উচিত?"
      

        
Logging trade-off — debug vs privacy + cost.

        
Pro logging:

        
    
          
  • Debug — "এই input-এ কী prediction".
    • 
          
  • Drift analysis — input distribution offline.
    • 
          
  • Replay — issue reproduce.
    • 
        

        
Con logging:

        
    
          
  • PII leak risk — name, NID, mobile.
    • 
          
  • Log volume explosion — cost.
    • 
          
  • GDPR/privacy regulation.
    • 
        

        
Compromise patterns:

        
    
          
  • Sample: 1% requests log full body.
    • 
          
  • Hash: PII hashed before log.
    • 
          
  • Schema only: log "got 5 features" not values.
    • 
          
  • Async log to data lake: raw log encrypted bucket; offline analysis.
    • 
        

        
Always log:

        
    
          
  • Request ID (correlation).
    • 
          
  • User ID (hashed).
    • 
          
  • Latency, status code.
    • 
          
  • Model version.
    • 
          
  • Error stack (failures only).
    • 
        

        
BD regulation note:

        
    
          
  • Bangladesh Bank — financial transaction log retain 5 years.
    • 
          
  • Personal Data Protection Act draft — data minimization.
    • 
          
  • Compliance team early consult.
    • 
        

        
মূল উপলব্ধি: Strategic logging — full body sample + structured metadata always. Privacy first; debug-ability next; cost mindful.

      

    

  


  

    
 অনুশীলন

    
    
      
  1. Build: উপরের template নিজের model-এ adapt করুন। localhost-এ run.
        
    uvicorn main:app --reload. http://localhost:8000/docs — Swagger UI auto-generated.
    
      
    2. 
      
  2. Load test: locust দিয়ে 100 RPS load — p99 latency কত?
        
    pip install locust; locustfile.py লিখুন; locust command চালান। Web UI-তে hatch rate set.
    
      
    3. 
      
  3. চিন্তা: এই service-এ Prometheus metrics যোগ করতে কী করবেন?
        
    prometheus-fastapi-instrumentator install + 1 line setup. /metrics endpoint auto-expose. Lesson 27-এ details.
    
      
    4. 
    

  


  

    
 আরও পড়ুন

    
  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ১৫ · Deployment patterns