পাঠ ১৭ · ৩৩-এর মধ্যে · মডিউল ৩
Home / AI Courses / MLOps / Triton

Triton Inference Server

NVIDIA Triton — multi-framework, GPU-optimized
৭ মিনিট পড়া উচ্চ · Advanced Config + Python

এই পাঠে যা শিখবেন

  • Triton model repository structure
  • config.pbtxt — model configuration
  • Dynamic batching — latency vs throughput tuning
  • FastAPI vs Triton — কখন কোনটা

১ · কেন Triton

FastAPI + sklearn ছোট model-এ সরল। কিন্তু GPU-heavy workload (BERT, ResNet, Stable Diffusion)-এ Triton dominant।

  • Multi-framework — TensorRT (fastest), ONNX, PyTorch, TF, OpenVINO।
  • Dynamic batching — multiple incoming requests merge করে GPU efficient।
  • Model ensemble — multi-step pipeline (preprocess → inference → postprocess) single request।
  • Multi-model — same GPU multiple models।
  • Concurrent execution — same model multiple instance parallel।

২ · Model repository structure

Bash · Repo layout 
model_repository/
├── bangla-sentiment-onnx/
│   ├── config.pbtxt
│   ├── 1/
│   │   └── model.onnx
│   └── 2/
│       └── model.onnx
├── image-classifier-tensorrt/
│   ├── config.pbtxt
│   ├── labels.txt
│   └── 1/
│       └── model.plan
└── llm-pytorch/
    ├── config.pbtxt
    └── 1/
        └── model.pt

    


    

      Model directory নাম = endpoint। Sub-directory version (1, 2)। config.pbtxt — model metadata + serving config।
    


    
৩ · config.pbtxt example


    

      

        
        Protobuf · config.pbtxt
      

      
name: "bangla-sentiment-onnx"
platform: "onnxruntime_onnx"
max_batch_size: 64

input [
  {
    name: "input_ids"
    data_type: TYPE_INT64
    dims: [ 128 ]
  },
  {
    name: "attention_mask"
    data_type: TYPE_INT64
    dims: [ 128 ]
  }
]

output [
  {
    name: "logits"
    data_type: TYPE_FP32
    dims: [ 3 ]
  }
]

dynamic_batching {
  preferred_batch_size: [ 4, 8, 16, 32 ]
  max_queue_delay_microseconds: 5000
}

instance_group [
  {
    count: 2
    kind: KIND_GPU
    gpus: [ 0 ]
  }
]

version_policy: { specific: { versions: [ 2 ] } }

      

        
        
      

      

    


    

      Critical configs: max_batch_size — server batch করার maximum। dynamic_batching.max_queue_delay — wait window। instance_group.count — concurrent execution।
    


    
৪ · Dynamic batching impact


    

      GPU inference per-request poor utilization। Batch-এ throughput dramatically up।
    

    
    
      
  • Batch size 1: GPU ~১০% utilized।
    • 
      
  • Batch size 16: GPU ~৭০% utilized।
    • 
      
  • Throughput: 5-10× higher।
    • 
    


    

      $$ \text{Effective\_latency} = \text{queue\_delay} + \text{batch\_inference\_time} $$
    


    

      Trade-off: queue_delay বাড়ালে batch বেশি (throughput up), কিন্তু individual request latency up।
    


    
৫ · Model ensemble


    

      একটি request-এ multi-step inference: tokenize → BERT → softmax → label। Each step Triton-এ separate model হিসেবে; ensemble একসাথে chain।
    


    
৬ · Triton client


    

      

        
        Python · Triton client
      

      
import tritonclient.http as httpclient
import numpy as np

client = httpclient.InferenceServerClient(url="triton:8000")

# input prepare
input_ids = np.array([[1, 2, 3, ..., 0]], dtype=np.int64)
attention_mask = np.ones_like(input_ids)

inputs = [
    httpclient.InferInput("input_ids", input_ids.shape, "INT64"),
    httpclient.InferInput("attention_mask", attention_mask.shape, "INT64"),
]
inputs[0].set_data_from_numpy(input_ids)
inputs[1].set_data_from_numpy(attention_mask)

outputs = [httpclient.InferRequestedOutput("logits")]

# inference
response = client.infer(
    model_name="bangla-sentiment-onnx",
    inputs=inputs,
    outputs=outputs,
)

logits = response.as_numpy("logits")
predictions = logits.argmax(axis=-1)
print(predictions)

      

        
        
      

      

    


    

      Triton HTTP/gRPC দু'টোই support। gRPC binary, lower latency। Python, C++, Java, Go client library available।
    


    
৭ · FastAPI vs Triton


    
    
      
  • FastAPI choose when:
        
      
          
    • Single sklearn / lightgbm model।
      • 
          
    • CPU-bound, light load।
      • 
          
    • Heavy preprocessing in Python।
      • 
          
    • Team Python-comfortable, no GPU expertise।
      • 
        
    
      
    • 
      
  • Triton choose when:
        
      
          
    • GPU-heavy (BERT, vision)।
      • 
          
    • Multi-model serving।
      • 
          
    • Throughput critical (1000+ RPS)।
      • 
          
    • TensorRT optimization wanted।
      • 
        
    
      
    • 
      
  • Hybrid:
        
      
          
    • FastAPI gateway → Triton inference backend।
      • 
          
    • FastAPI handles auth, validation, business logic।
      • 
          
    • Triton handles raw GPU inference।
      • 
        
    
      
    • 
    


    

      
        
        Triton dynamic batching — throughput multiplier

        
        Req 1
        
        Req 2
        
        Req 3
        
        Req 4

        
        Batch queue
        max_queue_delay = 5ms
        preferred_batch = 4, 8, 16
        Combined batch=4

        
        GPU inference
        batch=4 once
        ~25ms inference
        vs 4× single ~80ms
        throughput 3× ↑

        
        
        
        
        
      
      
Dynamic batching — multiple incoming request 5ms-window-এ collected, single GPU inference-এ। GPU utilization ও throughput dramatic boost।

    


    

      Bangladesh-এ Triton adoption growing — DL-heavy team-এ। Cloud GPU (AWS Mumbai g5 instances)-এ Triton standard। On-prem GPU shop-এ Triton Docker run trivial।
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      প্র ০১"Triton vs FastAPI — concrete decision criteria + scenario।"
      

        
Decision driven by workload type, not preference।

        
Triton wins:

        
    
          
  • Single GPU 500-2000 RPS BERT inference: dynamic batching 5-10× advantage।
    • 
          
  • Vision models (ResNet, Yolo) — TensorRT compile possible।
    • 
          
  • Multi-model — GPU memory share efficiency।
    • 
        

        
FastAPI wins:

        
    
          
  • Single sklearn model, 50 RPS — Triton overkill।
    • 
          
  • Heavy Python preprocessing (text cleaning, custom logic)।
    • 
          
  • Auth, rate limiting, business orchestration।
    • 
        

        
Hybrid (most common production):

        
    
          
  • FastAPI = API gateway, request validation, feature lookup, postprocess।
    • 
          
  • Triton = pure GPU inference backend।
    • 
          
  • FastAPI internally calls Triton via gRPC।
    • 
        

        
BD example:

        
    
          
  • Image-based product search Daraz: FastAPI receives upload → Triton (Yolo + ResNet ensemble) → FastAPI postprocess + return।
    • 
        

        
Cost:

        
    
          
  • Triton free, but GPU cost dominant।
    • 
          
  • Triton image larger (4-5 GB), FastAPI image slim (200 MB)।
    • 
        

        
মূল উপলব্ধি: Triton DL+GPU specialist। FastAPI generalist। 80% production hybrid। "FastAPI everything" — GPU under-utilized।

      

    


    

      প্র ০২"max_queue_delay tuning — কীভাবে?"
      

        
This is critical Triton tuning parameter।

        
What it does:

        
    
          
  • Server waits up to N microseconds for more requests to batch।
    • 
          
  • Batch fills early or timeout — execute।
    • 
        

        
Effects:

        
    
          
  • Higher delay = bigger batch = higher GPU efficiency = higher throughput।
    • 
          
  • But: individual request latency higher।
    • 
        

        
Tuning approach:

        
    
          
  • Start: max_queue_delay = your latency budget × 0.1 (e.g., 100ms budget → 10ms wait)।
    • 
          
  • Load test: vary delay, plot throughput vs p99 latency।
    • 
          
  • Sweet spot: max throughput within latency SLA।
    • 
        

        
Workload-specific:

        
    
          
  • High RPS (1000+): low delay (1-5ms) — batch fills naturally।
    • 
          
  • Medium RPS (100-500): medium delay (5-20ms)।
    • 
          
  • Low RPS: dynamic batching not very useful — just per-request inference।
    • 
        

        
preferred_batch_size:

        
    
          
  • List of sizes Triton prefers।
    • 
          
  • Common: [4, 8, 16, 32]।
    • 
          
  • Server batches to nearest preferred size।
    • 
        

        
Common pitfall:

        
    
          
  • Production batch size = max_batch_size — overshoots latency।
    • 
          
  • Ensure max_batch_size accommodates peak, not norm।
    • 
        

        
মূল উপলব্ধি: max_queue_delay = latency-throughput dial। Load test to find sweet spot। One-size-fits-all does not exist।

      

    


    

      প্র ০৩"TensorRT compile করা worth?"
      

        
TensorRT — NVIDIA's optimized inference engine।

        
Benefits:

        
    
          
  • 2-5× faster inference vs PyTorch।
    • 
          
  • FP16/INT8 quantization built-in।
    • 
          
  • Layer fusion, kernel auto-tuning।
    • 
        

        
Costs:

        
    
          
  • GPU-specific — A100 compiled engine doesn't run on T4।
    • 
          
  • Slow compile (5-30 min)।
    • 
          
  • Some PyTorch ops unsupported।
    • 
          
  • Debugging harder.
    • 
        

        
When worth:

        
    
          
  • Stable production model — frequent retrain rare।
    • 
          
  • Latency-critical — every ms matters।
    • 
          
  • Cost-critical — GPU hours expensive।
    • 
        

        
When skip:

        
    
          
  • Frequent model update — recompile overhead।
    • 
          
  • Custom layers — TensorRT support gap।
    • 
          
  • ONNX runtime usually 80% of TensorRT benefit, easier।
    • 
        

        
Practical workflow:

        
    
          
  • PyTorch → ONNX export।
    • 
          
  • ONNX → TensorRT compile (trtexec or polygraphy)।
    • 
          
  • Triton serve TensorRT engine।
    • 
        

        
BD context:

        
    
          
  • GPU expensive ($1.5-3/hour) — TensorRT 3× speedup → 3× cost saving।
    • 
          
  • For stable production — strong ROI।
    • 
        

        
মূল উপলব্ধি: TensorRT — production stable model-এ excellent। Frequent change-এ overhead exceed benefit। ONNX runtime safer middle ground।

      

    


    

      প্র ০৪"GPU concurrency — instance_group.count tuning।"
      

        
Multiple instance same model parallel — GPU memory + compute share।

        
What it does:

        
    
          
  • count=2: 2 copies of model on GPU; 2 inferences parallel।
    • 
          
  • Useful: model small, GPU underutilized in single instance।
    • 
        

        
Trade-offs:

        
    
          
  • Memory: each instance full model copy। Big LLM: maybe 1 instance fits।
    • 
          
  • Compute share: 2 instances share GPU; one slow other slower।
    • 
          
  • Latency variance: contention।
    • 
        

        
When increase count:

        
    
          
  • Small model (BERT-base 100MB)।
    • 
          
  • Mixed batch sizes — small batches don't saturate GPU।
    • 
        

        
When count=1:

        
    
          
  • Large LLM (GPT-style)।
    • 
          
  • Big batch saturates GPU fully।
    • 
        

        
Multi-GPU:

        
    
          
  • instance_group with multiple GPUs — model replicated across GPUs।
    • 
          
  • Triton load-balances across।
    • 
        

        
Tuning:

        
    
          
  • nvidia-smi monitor utilization।
    • 
          
  • Single instance: GPU 50% util — try count=2।
    • 
          
  • GPU 95% — single is enough।
    • 
        

        
মূল উপলব্ধি: instance_group.count — GPU utilization tuning dial। Default 1, increase if measure shows underutilization। Memory-aware।

      

    

  


  

    
 অনুশীলন

    
    
      
  1. Setup: Triton Docker run করুন; sample model serve।
        
    docker run --gpus=all -p8000:8000 -p8001:8001 -v /path/to/repo:/models nvcr.io/nvidia/tritonserver:24.01-py3 tritonserver --model-repository=/models
    
      
    2. 
      
  2. Convert: একটি sklearn / PyTorch model ONNX-এ convert; Triton-এ deploy।
        
    PyTorch: torch.onnx.export(model, ...). sklearn: skl2onnx. Place in repo, write config.pbtxt, restart Triton।
    
      
    3. 
      
  3. Tune: dynamic batching delay 1ms থেকে 50ms পর্যন্ত sweep — throughput-latency curve plot।
        
    Triton perf_analyzer tool ideal। perf_analyzer -m model_name --concurrency-range 1:32
    
      
    4. 
    

  


  

    
 আরও পড়ুন

    
  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ১৬ · FastAPI serving