পাঠ ২০ · ৩৩-এর মধ্যে · মডিউল ৩
Home / AI Courses / MLOps / Model optimization

Model optimization — quantization

Optimization — quantization, pruning, distillation
৭ মিনিট পড়া উচ্চ · Advanced Python

এই পাঠে যা শিখবেন

  • Quantization — types, trade-offs, calibration
  • Pruning ও knowledge distillation
  • ONNX export + runtime optimization
  • LLM quantization (GPTQ, AWQ, GGUF) overview

১ · কেন optimize

Production deployment-এ:

  • Latency: 100ms model → 30ms quantized — UX win.
  • Cost: GPU memory smaller → fewer GPUs needed.
  • Edge: mobile-এ 50 MB max — quantization essential.
  • Energy: battery, server cooling.

২ · Quantization types

  • FP32 → FP16: half precision, ~2× speedup, virtually no accuracy loss. Default for modern GPUs.
  • FP32 → BF16: brain float, similar to FP16 but better range. A100/H100 native.
  • FP32 → INT8: 4× memory + speed; calibration needed; accuracy drop 0.1-2%.
  • FP32 → INT4: 8× memory; accuracy drop measurable; LLM-popular.

৩ · Quantization-aware training (QAT) vs Post-training (PTQ)

  • PTQ: trained model directly quantize. Calibration data দিয়ে scale/zero-point compute. Easy, but accuracy drop sometimes.
  • QAT: training-time fake quantization simulate. More work, better accuracy.

৪ · PyTorch dynamic quantization example

Python · PyTorch quantization 
import torch
import torch.quantization

# Original FP32 model
model_fp32 = torch.load("model.pt").eval()

# 1. Dynamic quantization — easiest
# Quantize Linear/LSTM layers; activations dynamic
model_int8 = torch.quantization.quantize_dynamic(
    model_fp32,
    {torch.nn.Linear, torch.nn.LSTM},
    dtype=torch.qint8,
)

# Save smaller model
torch.save(model_int8.state_dict(), "model_int8.pt")

# Compare sizes
import os
fp32_size = os.path.getsize("model.pt") / 1024 / 1024
int8_size = os.path.getsize("model_int8.pt") / 1024 / 1024
print(f"FP32: {fp32_size:.1f} MB; INT8: {int8_size:.1f} MB ({fp32_size/int8_size:.1f}× smaller)")

# 2. Verify accuracy on validation set
import torch.utils.benchmark as benchmark
sample = torch.randn(1, 768)
t_fp32 = benchmark.Timer(stmt="m(x)", globals={"m": model_fp32, "x": sample}).timeit(100)
t_int8 = benchmark.Timer(stmt="m(x)", globals={"m": model_int8, "x": sample}).timeit(100)
print(f"FP32: {t_fp32.mean*1000:.2f}ms; INT8: {t_int8.mean*1000:.2f}ms")

    


    

      Dynamic quantization easiest — no calibration data needed. CPU-এ ভাল ROI. GPU-এ TensorRT static quantization preferred.
    


    
৫ · Pruning


    

      Many neural network weights ~zero — removing them no accuracy loss. Sparsity 70-90%-এ অনেক model-এ acceptable.
    

    
    
      
  • Magnitude pruning: smallest absolute value weights remove.
    • 
      
  • Structured pruning: entire neuron/channel remove — hardware-friendly.
    • 
      
  • Lottery ticket hypothesis: small sub-network exists at init that trains as well.
    • 
    


    
৬ · Knowledge distillation


    

      Large "teacher" model trained. Small "student" trained to match teacher's output (soft labels). DistilBERT — BERT-এর 40% size, 60% faster, 97% accuracy retained.
    


    
৭ · ONNX runtime optimization


    

      PyTorch/TF model → ONNX → ONNX Runtime. Cross-platform, optimized kernels, often 2-3× speedup.
    


    

      

        
        Python · ONNX export
      

      
import torch
import onnxruntime as ort
import numpy as np

# Export PyTorch → ONNX
sample_input = torch.randn(1, 3, 224, 224)
torch.onnx.export(
    model,
    sample_input,
    "model.onnx",
    input_names=["input"],
    output_names=["output"],
    dynamic_axes={"input": {0: "batch"}, "output": {0: "batch"}},
    opset_version=17,
)

# Inference with ONNX Runtime
session = ort.InferenceSession(
    "model.onnx",
    providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)
out = session.run(None, {"input": sample_input.numpy()})

# INT8 quantize ONNX model
from onnxruntime.quantization import quantize_dynamic, QuantType
quantize_dynamic(
    "model.onnx",
    "model_int8.onnx",
    weight_type=QuantType.QInt8,
)

      

        
        
      

      

    


    

      ONNX = framework-agnostic format. ONNX Runtime mobile/edge-এও চলে। Quantization built-in tools.
    


    
৮ · LLM quantization


    
    
      
  • GPTQ: post-training INT4 — minimal accuracy loss, popular for LLaMA/Mistral.
    • 
      
  • AWQ: Activation-aware Weight Quantization — better than GPTQ usually.
    • 
      
  • GGUF (formerly GGML): CPU-optimized, llama.cpp ecosystem. 4-bit-এ phone-এ চলে।
    • 
      
  • BitsAndBytes: easy 8-bit/4-bit loading via Hugging Face Transformers.
    • 
    


    

      LLaMA-7B FP16 = 14 GB; INT4 GPTQ = 4 GB. Single GPU fit possible.
    


    

      
        
        Quantization — accuracy vs size/speed trade-off

        
        FP32
        100% accuracy
        100 MB
        100ms latency
        Default
        no optimization

        
        FP16/BF16
        ~99.9% accuracy
        50 MB (2× smaller)
        ~50ms (2× faster)
        Easy win
        no calibration

        
        INT8
        ~99% accuracy
        25 MB (4× smaller)
        ~30ms (3× faster)
        Calibration needed
        Production sweet spot

        
        INT4
        ~95-98% accuracy
        12 MB (8× smaller)
        ~20ms
        LLM popular
        GPTQ, AWQ

        Lower precision = more compression + speed; accuracy slowly degrades.
        Validation always required after quantization.
      
      
Quantization spectrum — FP32 baseline; FP16 free win; INT8 production sweet spot; INT4 for LLM with calibration.

    


    

      Bangladesh GPU-cost-conscious — quantization adoption growing. Bangla LLM serving INT4 typical. Mobile app on-device — INT8 essential.
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      প্র ০১"INT8 calibration data কী, কতটুকু লাগে?"
      

        
INT8 quantization-এ activation distribution-এর scale/zero-point compute করতে representative data দরকার।

        
Calibration purpose:

        
    
          
  • Float-এর range [min, max] discover.
    • 
          
  • INT8 [-128, 127] map করার scale calculate.
    • 
          
  • Per-layer or per-channel.
    • 
        

        
How much data:

        
    
          
  • Typical: ১০০-১০০০ samples enough.
    • 
          
  • More — diminishing returns.
    • 
          
  • Quality matter more than quantity — distribution-representative.
    • 
        

        
Strategies:

        
    
          
  • MinMax: simplest — observed min/max use.
    • 
          
  • Entropy/KL: minimize information loss; better accuracy.
    • 
          
  • Percentile: ignore outliers (e.g., 99.99 percentile).
    • 
        

        
BD context — Bangla NLP:

        
    
          
  • Bangla text variation high; cover formal + informal + dialect.
    • 
          
  • Calibration set 500 sample diverse texts.
    • 
        

        
Pitfall:

        
    
          
  • Calibration on training data — production drift mismatch.
    • 
          
  • Better: real production sample (recent week).
    • 
        

        
মূল উপলব্ধি: Calibration accuracy-quantized model-এর key. Production-representative data; few hundred sample enough; quality matter most.

      

    


    

      প্র ০২"Mobile (Bangla keyboard) — কোন quantization?"
      

        
On-device deployment unique constraints.

        
Constraints:

        
    
          
  • App size — Bangla keyboard 30 MB target.
    • 
          
  • RAM — budget Android 2-4 GB; model 50 MB max.
    • 
          
  • Inference speed — type during keystroke, < 50ms required.
    • 
          
  • Battery — heavy compute drains.
    • 
        

        
Approach:

        
    
          
  • Distill large LM → small student.
    • 
          
  • INT8 quantize via TF Lite.
    • 
          
  • Model size 5-15 MB target.
    • 
          
  • Latency 10-30ms typical.
    • 
        

        
Tools:

        
    
          
  • TF Lite — Android first-class.
    • 
          
  • TF Lite GPU delegate — newer phones GPU/NPU acceleration.
    • 
          
  • ONNX Runtime mobile — alternative.
    • 
          
  • MediaPipe — Google's prebuilt blocks.
    • 
        

        
Bangla-specific:

        
    
          
  • Tokenization — Bangla ZWJ/ZWNJ tricky on small models.
    • 
          
  • Character-level often better than subword for low-resource.
    • 
          
  • Embedding sharing — input/output tied.
    • 
        

        
Update strategy:

        
    
          
  • App store update slow; over-the-air model update via API.
    • 
          
  • Fallback to bundled if download fails.
    • 
        

        
মূল উপলব্ধি: Mobile = aggressive optimization mandatory. Distill + INT8 + TF Lite. Bangla-specific tokenization care.

      

    


    

      প্র ০৩"Pruning vs distillation — কোনটা কখন?"
      

        
দু'টি orthogonal techniques, complementary.

        
Pruning:

        
    
          
  • Existing model weights remove.
    • 
          
  • No retraining needed (sometimes fine-tune).
    • 
          
  • Architecture intact, sparsity introduced.
    • 
          
  • Hardware support varies (NVIDIA Ampere structured sparsity hardware-accelerated).
    • 
        

        
Distillation:

        
    
          
  • New smaller architecture from scratch trained mimic teacher.
    • 
          
  • Architecture-level optimization.
    • 
          
  • Significant training cost.
    • 
          
  • Better accuracy retention generally.
    • 
        

        
Combine:

        
    
          
  • Distill first (architecture compression).
    • 
          
  • Then quantize student (precision compression).
    • 
          
  • Then prune within layers (sparsity compression).
    • 
          
  • Compounded gains.
    • 
        

        
BD context:

        
    
          
  • Distillation training-cost-heavy — only invest if model used long-term.
    • 
          
  • Pruning quick win for existing models.
    • 
          
  • Quantization easiest first try.
    • 
        

        
মূল উপলব্ধি: Order: quantize → prune → distill (cost ascending). Stop at sufficient gain.

      

    


    

      প্র ০৪"LLM 4-bit quantization production-safe?"
      

        
LLM 4-bit (GPTQ/AWQ) — surprising effective, but caveats.

        
Accuracy retention:

        
    
          
  • LLaMA-2 7B FP16 vs 4-bit GPTQ — perplexity ~1% increase.
    • 
          
  • Most benchmarks (MMLU, HellaSwag) — within 1-2%.
    • 
          
  • "User won't notice" generally.
    • 
        

        
Failure modes:

        
    
          
  • Long-tail rare token — worse accuracy.
    • 
          
  • Code generation — sometimes degraded.
    • 
          
  • Multi-language — high-resource OK, low-resource (Bangla) more drop.
    • 
        

        
Production validation:

        
    
          
  • Domain-specific eval set.
    • 
          
  • Compare FP16 vs 4-bit side-by-side.
    • 
          
  • If user-facing — A/B test.
    • 
        

        
Methods:

        
    
          
  • GPTQ — original; calibration data needed.
    • 
          
  • AWQ — newer, better; activation-aware.
    • 
          
  • GGUF (llama.cpp) — CPU-optimized; phone-deployable.
    • 
          
  • BitsAndBytes 4-bit — easy HF integration.
    • 
        

        
Hardware support:

        
    
          
  • 4-bit native: H100, latest GPU.
    • 
          
  • Older GPU: 4-bit-from-INT8 emulation; less benefit.
    • 
        

        
BD context — Bangla LLM:

        
    
          
  • Self-hosted Bangla LLM — 4-bit GPU memory enable.
    • 
          
  • Validation Bangla benchmark (LLM evaluation Lesson 30) essential.
    • 
          
  • Quality drop tolerable for cost saving usually.
    • 
        

        
মূল উপলব্ধি: LLM 4-bit — generally production-safe with validation. Bangla underspecified low-resource — extra care. Not blanket apply; verify.

      

    

  


  

    
 অনুশীলন

    
    
      
  1. Quantize: একটি sample sklearn/PyTorch model FP16/INT8-এ convert. Latency + accuracy compare.
        
    উপরের code use. FP32 vs FP16 vs INT8 — table form.
    
      
    2. 
      
  2. ONNX: Same model ONNX export + ONNX Runtime দিয়ে inference. Speedup measure.
        
    torch.onnx.export; onnxruntime.InferenceSession. Generally 1.5-3× faster.
    
      
    3. 
      
  3. চিন্তা: bKash low-end phone fraud check — optimization stack design করুন।
        
    
          
      
            
    • Distill heavy server model → small mobile model.
      • 
            
    • INT8 quantize via TF Lite.
      • 
            
    • Phone GPU delegate enable.
      • 
            
    • Server fallback for low-confidence cases.
      • 
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন

    
  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ১৯ · Batch vs online