পাঠ ১৭ · ৩৩-এর মধ্যে · মডিউল ৩
Home / AI Courses / MLOps / Triton

Triton Inference Server

NVIDIA Triton — multi-framework, GPU-optimized
৭ মিনিট পড়া উচ্চ · Advanced Config + Python

এই পাঠে যা শিখবেন

  • Triton model repository structure
  • config.pbtxt — model configuration
  • Dynamic batching — latency vs throughput tuning
  • FastAPI vs Triton — কখন কোনটা

১ · কেন Triton

FastAPI + sklearn ছোট model-এ সরল। কিন্তু GPU-heavy workload (BERT, ResNet, Stable Diffusion)-এ Triton dominant.

  • Multi-framework — TensorRT (fastest), ONNX, PyTorch, TF, OpenVINO.
  • Dynamic batching — multiple incoming requests merge করে GPU efficient.
  • Model ensemble — multi-step pipeline (preprocess → inference → postprocess) single request.
  • Multi-model — same GPU multiple models.
  • Concurrent execution — same model multiple instance parallel.

২ · Model repository structure

Bash · Repo layout 
model_repository/
├── bangla-sentiment-onnx/
│   ├── config.pbtxt
│   ├── 1/
│   │   └── model.onnx
│   └── 2/
│       └── model.onnx
├── image-classifier-tensorrt/
│   ├── config.pbtxt
│   ├── labels.txt
│   └── 1/
│       └── model.plan
└── llm-pytorch/
    ├── config.pbtxt
    └── 1/
        └── model.pt

    


    

      Model directory নাম = endpoint. Sub-directory version (1, 2). config.pbtxt — model metadata + serving config.
    


    
৩ · config.pbtxt example


    

      

        
        Protobuf · config.pbtxt
      

      
name: "bangla-sentiment-onnx"
platform: "onnxruntime_onnx"
max_batch_size: 64

input [
  {
    name: "input_ids"
    data_type: TYPE_INT64
    dims: [ 128 ]
  },
  {
    name: "attention_mask"
    data_type: TYPE_INT64
    dims: [ 128 ]
  }
]

output [
  {
    name: "logits"
    data_type: TYPE_FP32
    dims: [ 3 ]
  }
]

dynamic_batching {
  preferred_batch_size: [ 4, 8, 16, 32 ]
  max_queue_delay_microseconds: 5000
}

instance_group [
  {
    count: 2
    kind: KIND_GPU
    gpus: [ 0 ]
  }
]

version_policy: { specific: { versions: [ 2 ] } }

      

        
        
      

      

    


    

      Critical configs: max_batch_size — server batch করার maximum. dynamic_batching.max_queue_delay — wait window. instance_group.count — concurrent execution.
    


    
৪ · Dynamic batching impact


    

      GPU inference per-request poor utilization. Batch-এ throughput dramatically up.
    

    
    
      
  • Batch size 1: GPU ~১০% utilized.
    • 
      
  • Batch size 16: GPU ~৭০% utilized.
    • 
      
  • Throughput: 5-10× higher.
    • 
    


    

      $$ \text{Effective\_latency} = \text{queue\_delay} + \text{batch\_inference\_time} $$
    


    

      Trade-off: queue_delay বাড়ালে batch বেশি (throughput up), কিন্তু individual request latency up.
    


    
৫ · Model ensemble


    

      একটি request-এ multi-step inference: tokenize → BERT → softmax → label. Each step Triton-এ separate model হিসেবে; ensemble একসাথে chain.
    


    
৬ · Triton client


    

      

        
        Python · Triton client
      

      
import tritonclient.http as httpclient
import numpy as np

client = httpclient.InferenceServerClient(url="triton:8000")

# input prepare
input_ids = np.array([[1, 2, 3, ..., 0]], dtype=np.int64)
attention_mask = np.ones_like(input_ids)

inputs = [
    httpclient.InferInput("input_ids", input_ids.shape, "INT64"),
    httpclient.InferInput("attention_mask", attention_mask.shape, "INT64"),
]
inputs[0].set_data_from_numpy(input_ids)
inputs[1].set_data_from_numpy(attention_mask)

outputs = [httpclient.InferRequestedOutput("logits")]

# inference
response = client.infer(
    model_name="bangla-sentiment-onnx",
    inputs=inputs,
    outputs=outputs,
)

logits = response.as_numpy("logits")
predictions = logits.argmax(axis=-1)
print(predictions)

      

        
        
      

      

    


    

      Triton HTTP/gRPC দু'টোই support. gRPC binary, lower latency. Python, C++, Java, Go client library available.
    


    
৭ · FastAPI vs Triton


    
    
      
  • FastAPI choose when:
        
      
          
    • Single sklearn / lightgbm model.
      • 
          
    • CPU-bound, light load.
      • 
          
    • Heavy preprocessing in Python.
      • 
          
    • Team Python-comfortable, no GPU expertise.
      • 
        
    
      
    • 
      
  • Triton choose when:
        
      
          
    • GPU-heavy (BERT, vision).
      • 
          
    • Multi-model serving.
      • 
          
    • Throughput critical (1000+ RPS).
      • 
          
    • TensorRT optimization wanted.
      • 
        
    
      
    • 
      
  • Hybrid:
        
      
          
    • FastAPI gateway → Triton inference backend.
      • 
          
    • FastAPI handles auth, validation, business logic.
      • 
          
    • Triton handles raw GPU inference.
      • 
        
    
      
    • 
    


    

      
        
        Triton dynamic batching — throughput multiplier

        
        Req 1
        
        Req 2
        
        Req 3
        
        Req 4

        
        Batch queue
        max_queue_delay = 5ms
        preferred_batch = 4, 8, 16
        Combined batch=4

        
        GPU inference
        batch=4 once
        ~25ms inference
        vs 4× single ~80ms
        throughput 3× ↑

        
        
        
        
        
      
      
Dynamic batching — multiple incoming request 5ms-window-এ collected, single GPU inference-এ। GPU utilization ও throughput dramatic boost.

    


    

      Bangladesh-এ Triton adoption growing — DL-heavy team-এ। Cloud GPU (AWS Mumbai g5 instances)-এ Triton standard. On-prem GPU shop-এ Triton Docker run trivial.
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      প্র ০১"Triton vs FastAPI — concrete decision criteria + scenario."
      

        
Decision driven by workload type, not preference.

        
Triton wins:

        
    
          
  • Single GPU 500-2000 RPS BERT inference: dynamic batching 5-10× advantage.
    • 
          
  • Vision models (ResNet, Yolo) — TensorRT compile possible.
    • 
          
  • Multi-model — GPU memory share efficiency.
    • 
        

        
FastAPI wins:

        
    
          
  • Single sklearn model, 50 RPS — Triton overkill.
    • 
          
  • Heavy Python preprocessing (text cleaning, custom logic).
    • 
          
  • Auth, rate limiting, business orchestration.
    • 
        

        
Hybrid (most common production):

        
    
          
  • FastAPI = API gateway, request validation, feature lookup, postprocess.
    • 
          
  • Triton = pure GPU inference backend.
    • 
          
  • FastAPI internally calls Triton via gRPC.
    • 
        

        
BD example:

        
    
          
  • Image-based product search Daraz: FastAPI receives upload → Triton (Yolo + ResNet ensemble) → FastAPI postprocess + return.
    • 
        

        
Cost:

        
    
          
  • Triton free, but GPU cost dominant.
    • 
          
  • Triton image larger (4-5 GB), FastAPI image slim (200 MB).
    • 
        

        
মূল উপলব্ধি: Triton DL+GPU specialist. FastAPI generalist. 80% production hybrid. "FastAPI everything" — GPU under-utilized.

      

    


    

      প্র ০২"max_queue_delay tuning — কীভাবে?"
      

        
This is critical Triton tuning parameter.

        
What it does:

        
    
          
  • Server waits up to N microseconds for more requests to batch.
    • 
          
  • Batch fills early or timeout — execute.
    • 
        

        
Effects:

        
    
          
  • Higher delay = bigger batch = higher GPU efficiency = higher throughput.
    • 
          
  • But: individual request latency higher.
    • 
        

        
Tuning approach:

        
    
          
  • Start: max_queue_delay = your latency budget × 0.1 (e.g., 100ms budget → 10ms wait).
    • 
          
  • Load test: vary delay, plot throughput vs p99 latency.
    • 
          
  • Sweet spot: max throughput within latency SLA.
    • 
        

        
Workload-specific:

        
    
          
  • High RPS (1000+): low delay (1-5ms) — batch fills naturally.
    • 
          
  • Medium RPS (100-500): medium delay (5-20ms).
    • 
          
  • Low RPS: dynamic batching not very useful — just per-request inference.
    • 
        

        
preferred_batch_size:

        
    
          
  • List of sizes Triton prefers.
    • 
          
  • Common: [4, 8, 16, 32].
    • 
          
  • Server batches to nearest preferred size.
    • 
        

        
Common pitfall:

        
    
          
  • Production batch size = max_batch_size — overshoots latency.
    • 
          
  • Ensure max_batch_size accommodates peak, not norm.
    • 
        

        
মূল উপলব্ধি: max_queue_delay = latency-throughput dial. Load test to find sweet spot. One-size-fits-all does not exist.

      

    


    

      প্র ০৩"TensorRT compile করা worth?"
      

        
TensorRT — NVIDIA's optimized inference engine.

        
Benefits:

        
    
          
  • 2-5× faster inference vs PyTorch.
    • 
          
  • FP16/INT8 quantization built-in.
    • 
          
  • Layer fusion, kernel auto-tuning.
    • 
        

        
Costs:

        
    
          
  • GPU-specific — A100 compiled engine doesn't run on T4.
    • 
          
  • Slow compile (5-30 min).
    • 
          
  • Some PyTorch ops unsupported.
    • 
          
  • Debugging harder.
    • 
        

        
When worth:

        
    
          
  • Stable production model — frequent retrain rare.
    • 
          
  • Latency-critical — every ms matters.
    • 
          
  • Cost-critical — GPU hours expensive.
    • 
        

        
When skip:

        
    
          
  • Frequent model update — recompile overhead.
    • 
          
  • Custom layers — TensorRT support gap.
    • 
          
  • ONNX runtime usually 80% of TensorRT benefit, easier.
    • 
        

        
Practical workflow:

        
    
          
  • PyTorch → ONNX export.
    • 
          
  • ONNX → TensorRT compile (trtexec or polygraphy).
    • 
          
  • Triton serve TensorRT engine.
    • 
        

        
BD context:

        
    
          
  • GPU expensive ($1.5-3/hour) — TensorRT 3× speedup → 3× cost saving.
    • 
          
  • For stable production — strong ROI.
    • 
        

        
মূল উপলব্ধি: TensorRT — production stable model-এ excellent. Frequent change-এ overhead exceed benefit. ONNX runtime safer middle ground.

      

    


    

      প্র ০৪"GPU concurrency — instance_group.count tuning."
      

        
Multiple instance same model parallel — GPU memory + compute share.

        
What it does:

        
    
          
  • count=2: 2 copies of model on GPU; 2 inferences parallel.
    • 
          
  • Useful: model small, GPU underutilized in single instance.
    • 
        

        
Trade-offs:

        
    
          
  • Memory: each instance full model copy. Big LLM: maybe 1 instance fits.
    • 
          
  • Compute share: 2 instances share GPU; one slow other slower.
    • 
          
  • Latency variance: contention.
    • 
        

        
When increase count:

        
    
          
  • Small model (BERT-base 100MB).
    • 
          
  • Mixed batch sizes — small batches don't saturate GPU.
    • 
        

        
When count=1:

        
    
          
  • Large LLM (GPT-style).
    • 
          
  • Big batch saturates GPU fully.
    • 
        

        
Multi-GPU:

        
    
          
  • instance_group with multiple GPUs — model replicated across GPUs.
    • 
          
  • Triton load-balances across.
    • 
        

        
Tuning:

        
    
          
  • nvidia-smi monitor utilization.
    • 
          
  • Single instance: GPU 50% util — try count=2.
    • 
          
  • GPU 95% — single is enough.
    • 
        

        
মূল উপলব্ধি: instance_group.count — GPU utilization tuning dial. Default 1, increase if measure shows underutilization. Memory-aware.

      

    

  


  

    
 অনুশীলন

    
    
      
  1. Setup: Triton Docker run করুন; sample model serve.
        
    docker run --gpus=all -p8000:8000 -p8001:8001 -v /path/to/repo:/models nvcr.io/nvidia/tritonserver:24.01-py3 tritonserver --model-repository=/models
    
      
    2. 
      
  2. Convert: একটি sklearn / PyTorch model ONNX-এ convert; Triton-এ deploy.
        
    PyTorch: torch.onnx.export(model, ...). sklearn: skl2onnx. Place in repo, write config.pbtxt, restart Triton.
    
      
    3. 
      
  3. Tune: dynamic batching delay 1ms থেকে 50ms পর্যন্ত sweep — throughput-latency curve plot.
        
    Triton perf_analyzer tool ideal. perf_analyzer -m model_name --concurrency-range 1:32
    
      
    4. 
    

  


  

    
 আরও পড়ুন

    
  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ১৬ · FastAPI serving