পাঠ ২০ · ৩৩-এর মধ্যে · মডিউল ৩
Home / AI Courses / MLOps / Model optimization

Model optimization — quantization

Optimization — quantization, pruning, distillation
৭ মিনিট পড়া উচ্চ · Advanced Python

এই পাঠে যা শিখবেন

  • Quantization — types, trade-offs, calibration
  • Pruning ও knowledge distillation
  • ONNX export + runtime optimization
  • LLM quantization (GPTQ, AWQ, GGUF) overview

১ · কেন optimize

Production deployment-এ:

  • Latency: 100ms model → 30ms quantized — UX win।
  • Cost: GPU memory smaller → fewer GPUs needed।
  • Edge: mobile-এ 50 MB max — quantization essential।
  • Energy: battery, server cooling।

২ · Quantization types

  • FP32 → FP16: half precision, ~2× speedup, virtually no accuracy loss। Default for modern GPUs।
  • FP32 → BF16: brain float, similar to FP16 but better range। A100/H100 native।
  • FP32 → INT8: 4× memory + speed; calibration needed; accuracy drop 0.1-2%।
  • FP32 → INT4: 8× memory; accuracy drop measurable; LLM-popular।

৩ · Quantization-aware training (QAT) vs Post-training (PTQ)

  • PTQ: trained model directly quantize। Calibration data দিয়ে scale/zero-point compute। Easy, but accuracy drop sometimes।
  • QAT: training-time fake quantization simulate। More work, better accuracy।

৪ · PyTorch dynamic quantization example

Python · PyTorch quantization 
import torch
import torch.quantization

# Original FP32 model
model_fp32 = torch.load("model.pt").eval()

# 1. Dynamic quantization — easiest
# Quantize Linear/LSTM layers; activations dynamic
model_int8 = torch.quantization.quantize_dynamic(
    model_fp32,
    {torch.nn.Linear, torch.nn.LSTM},
    dtype=torch.qint8,
)

# Save smaller model
torch.save(model_int8.state_dict(), "model_int8.pt")

# Compare sizes
import os
fp32_size = os.path.getsize("model.pt") / 1024 / 1024
int8_size = os.path.getsize("model_int8.pt") / 1024 / 1024
print(f"FP32: {fp32_size:.1f} MB; INT8: {int8_size:.1f} MB ({fp32_size/int8_size:.1f}× smaller)")

# 2. Verify accuracy on validation set
import torch.utils.benchmark as benchmark
sample = torch.randn(1, 768)
t_fp32 = benchmark.Timer(stmt="m(x)", globals={"m": model_fp32, "x": sample}).timeit(100)
t_int8 = benchmark.Timer(stmt="m(x)", globals={"m": model_int8, "x": sample}).timeit(100)
print(f"FP32: {t_fp32.mean*1000:.2f}ms; INT8: {t_int8.mean*1000:.2f}ms")

    


    

      Dynamic quantization easiest — no calibration data needed। CPU-এ ভাল ROI। GPU-এ TensorRT static quantization preferred।
    


    
৫ · Pruning


    

      Many neural network weights ~zero — removing them no accuracy loss। Sparsity 70-90%-এ অনেক model-এ acceptable।
    

    
    
      
  • Magnitude pruning: smallest absolute value weights remove।
    • 
      
  • Structured pruning: entire neuron/channel remove — hardware-friendly।
    • 
      
  • Lottery ticket hypothesis: small sub-network exists at init that trains as well।
    • 
    


    
৬ · Knowledge distillation


    

      Large "teacher" model trained। Small "student" trained to match teacher's output (soft labels)। DistilBERT — BERT-এর 40% size, 60% faster, 97% accuracy retained।
    


    
৭ · ONNX runtime optimization


    

      PyTorch/TF model → ONNX → ONNX Runtime। Cross-platform, optimized kernels, often 2-3× speedup।
    


    

      

        
        Python · ONNX export
      

      
import torch
import onnxruntime as ort
import numpy as np

# Export PyTorch → ONNX
sample_input = torch.randn(1, 3, 224, 224)
torch.onnx.export(
    model,
    sample_input,
    "model.onnx",
    input_names=["input"],
    output_names=["output"],
    dynamic_axes={"input": {0: "batch"}, "output": {0: "batch"}},
    opset_version=17,
)

# Inference with ONNX Runtime
session = ort.InferenceSession(
    "model.onnx",
    providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)
out = session.run(None, {"input": sample_input.numpy()})

# INT8 quantize ONNX model
from onnxruntime.quantization import quantize_dynamic, QuantType
quantize_dynamic(
    "model.onnx",
    "model_int8.onnx",
    weight_type=QuantType.QInt8,
)

      

        
        
      

      

    


    

      ONNX = framework-agnostic format। ONNX Runtime mobile/edge-এও চলে। Quantization built-in tools।
    


    
৮ · LLM quantization


    
    
      
  • GPTQ: post-training INT4 — minimal accuracy loss, popular for LLaMA/Mistral।
    • 
      
  • AWQ: Activation-aware Weight Quantization — better than GPTQ usually।
    • 
      
  • GGUF (formerly GGML): CPU-optimized, llama.cpp ecosystem। 4-bit-এ phone-এ চলে।
    • 
      
  • BitsAndBytes: easy 8-bit/4-bit loading via Hugging Face Transformers।
    • 
    


    

      LLaMA-7B FP16 = 14 GB; INT4 GPTQ = 4 GB। Single GPU fit possible।
    


    

      
        
        Quantization — accuracy vs size/speed trade-off

        
        FP32
        100% accuracy
        100 MB
        100ms latency
        Default
        no optimization

        
        FP16/BF16
        ~99.9% accuracy
        50 MB (2× smaller)
        ~50ms (2× faster)
        Easy win
        no calibration

        
        INT8
        ~99% accuracy
        25 MB (4× smaller)
        ~30ms (3× faster)
        Calibration needed
        Production sweet spot

        
        INT4
        ~95-98% accuracy
        12 MB (8× smaller)
        ~20ms
        LLM popular
        GPTQ, AWQ

        Lower precision = more compression + speed; accuracy slowly degrades।
        Validation always required after quantization।
      
      
Quantization spectrum — FP32 baseline; FP16 free win; INT8 production sweet spot; INT4 for LLM with calibration।

    


    

      Bangladesh GPU-cost-conscious — quantization adoption growing। Bangla LLM serving INT4 typical। Mobile app on-device — INT8 essential।
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      প্র ০১"INT8 calibration data কী, কতটুকু লাগে?"
      

        
INT8 quantization-এ activation distribution-এর scale/zero-point compute করতে representative data দরকার।

        
Calibration purpose:

        
    
          
  • Float-এর range [min, max] discover।
    • 
          
  • INT8 [-128, 127] map করার scale calculate।
    • 
          
  • Per-layer or per-channel।
    • 
        

        
How much data:

        
    
          
  • Typical: ১০০-১০০০ samples enough।
    • 
          
  • More — diminishing returns।
    • 
          
  • Quality matter more than quantity — distribution-representative।
    • 
        

        
Strategies:

        
    
          
  • MinMax: simplest — observed min/max use।
    • 
          
  • Entropy/KL: minimize information loss; better accuracy।
    • 
          
  • Percentile: ignore outliers (e.g., 99.99 percentile)।
    • 
        

        
BD context — Bangla NLP:

        
    
          
  • Bangla text variation high; cover formal + informal + dialect।
    • 
          
  • Calibration set 500 sample diverse texts।
    • 
        

        
Pitfall:

        
    
          
  • Calibration on training data — production drift mismatch।
    • 
          
  • Better: real production sample (recent week)।
    • 
        

        
মূল উপলব্ধি: Calibration accuracy-quantized model-এর key। Production-representative data; few hundred sample enough; quality matter most।

      

    


    

      প্র ০২"Mobile (Bangla keyboard) — কোন quantization?"
      

        
On-device deployment unique constraints।

        
Constraints:

        
    
          
  • App size — Bangla keyboard 30 MB target।
    • 
          
  • RAM — budget Android 2-4 GB; model 50 MB max।
    • 
          
  • Inference speed — type during keystroke, < 50ms required।
    • 
          
  • Battery — heavy compute drains।
    • 
        

        
Approach:

        
    
          
  • Distill large LM → small student।
    • 
          
  • INT8 quantize via TF Lite।
    • 
          
  • Model size 5-15 MB target।
    • 
          
  • Latency 10-30ms typical।
    • 
        

        
Tools:

        
    
          
  • TF Lite — Android first-class।
    • 
          
  • TF Lite GPU delegate — newer phones GPU/NPU acceleration।
    • 
          
  • ONNX Runtime mobile — alternative।
    • 
          
  • MediaPipe — Google's prebuilt blocks।
    • 
        

        
Bangla-specific:

        
    
          
  • Tokenization — Bangla ZWJ/ZWNJ tricky on small models।
    • 
          
  • Character-level often better than subword for low-resource।
    • 
          
  • Embedding sharing — input/output tied।
    • 
        

        
Update strategy:

        
    
          
  • App store update slow; over-the-air model update via API।
    • 
          
  • Fallback to bundled if download fails।
    • 
        

        
মূল উপলব্ধি: Mobile = aggressive optimization mandatory। Distill + INT8 + TF Lite। Bangla-specific tokenization care।

      

    


    

      প্র ০৩"Pruning vs distillation — কোনটা কখন?"
      

        
দু'টি orthogonal techniques, complementary।

        
Pruning:

        
    
          
  • Existing model weights remove।
    • 
          
  • No retraining needed (sometimes fine-tune)।
    • 
          
  • Architecture intact, sparsity introduced।
    • 
          
  • Hardware support varies (NVIDIA Ampere structured sparsity hardware-accelerated)।
    • 
        

        
Distillation:

        
    
          
  • New smaller architecture from scratch trained mimic teacher।
    • 
          
  • Architecture-level optimization।
    • 
          
  • Significant training cost।
    • 
          
  • Better accuracy retention generally।
    • 
        

        
Combine:

        
    
          
  • Distill first (architecture compression)।
    • 
          
  • Then quantize student (precision compression)।
    • 
          
  • Then prune within layers (sparsity compression)।
    • 
          
  • Compounded gains।
    • 
        

        
BD context:

        
    
          
  • Distillation training-cost-heavy — only invest if model used long-term।
    • 
          
  • Pruning quick win for existing models।
    • 
          
  • Quantization easiest first try।
    • 
        

        
মূল উপলব্ধি: Order: quantize → prune → distill (cost ascending)। Stop at sufficient gain।

      

    


    

      প্র ০৪"LLM 4-bit quantization production-safe?"
      

        
LLM 4-bit (GPTQ/AWQ) — surprising effective, but caveats।

        
Accuracy retention:

        
    
          
  • LLaMA-2 7B FP16 vs 4-bit GPTQ — perplexity ~1% increase।
    • 
          
  • Most benchmarks (MMLU, HellaSwag) — within 1-2%।
    • 
          
  • "User won't notice" generally।
    • 
        

        
Failure modes:

        
    
          
  • Long-tail rare token — worse accuracy।
    • 
          
  • Code generation — sometimes degraded।
    • 
          
  • Multi-language — high-resource OK, low-resource (Bangla) more drop।
    • 
        

        
Production validation:

        
    
          
  • Domain-specific eval set।
    • 
          
  • Compare FP16 vs 4-bit side-by-side।
    • 
          
  • If user-facing — A/B test।
    • 
        

        
Methods:

        
    
          
  • GPTQ — original; calibration data needed।
    • 
          
  • AWQ — newer, better; activation-aware।
    • 
          
  • GGUF (llama.cpp) — CPU-optimized; phone-deployable।
    • 
          
  • BitsAndBytes 4-bit — easy HF integration।
    • 
        

        
Hardware support:

        
    
          
  • 4-bit native: H100, latest GPU।
    • 
          
  • Older GPU: 4-bit-from-INT8 emulation; less benefit।
    • 
        

        
BD context — Bangla LLM:

        
    
          
  • Self-hosted Bangla LLM — 4-bit GPU memory enable।
    • 
          
  • Validation Bangla benchmark (LLM evaluation Lesson 30) essential।
    • 
          
  • Quality drop tolerable for cost saving usually।
    • 
        

        
মূল উপলব্ধি: LLM 4-bit — generally production-safe with validation। Bangla underspecified low-resource — extra care। Not blanket apply; verify।

      

    

  


  

    
 অনুশীলন

    
    
      
  1. Quantize: একটি sample sklearn/PyTorch model FP16/INT8-এ convert। Latency + accuracy compare।
        
    উপরের code use। FP32 vs FP16 vs INT8 — table form।
    
      
    2. 
      
  2. ONNX: Same model ONNX export + ONNX Runtime দিয়ে inference। Speedup measure।
        
    torch.onnx.export; onnxruntime.InferenceSession। Generally 1.5-3× faster.
    
      
    3. 
      
  3. চিন্তা: bKash low-end phone fraud check — optimization stack design করুন।
        
    
          
      
            
    • Distill heavy server model → small mobile model।
      • 
            
    • INT8 quantize via TF Lite।
      • 
            
    • Phone GPU delegate enable।
      • 
            
    • Server fallback for low-confidence cases।
      • 
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন

    
  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ১৯ · Batch vs online