পাঠ ২৫ · ৩৫-এর মধ্যে · মডিউল ৪
Home / AI Courses / Computer Vision / ViT

Vision Transformer (ViT)

Vision Transformer — image as sequence of patches
৮ মিনিট পড়া উচ্চ · Advanced PyTorch কোডসহ

এই পাঠে যা শিখবেন

  • ViT architecture step-by-step
  • Patch embedding, position embedding
  • Self-attention CV-তে
  • ViT vs CNN trade-off

১ · ViT-এর big idea

Transformer (২০১৭) NLP-এ revolution। ২০২০ পর্যন্ত CV-তে CNN dominant। Dosovitskiy et al. — "We do not need convolution at all"।

Approach: image-কে non-overlapping patches-এ ভাগ → প্রতিটি patch = "word"। Standard NLP transformer apply।

কেন্দ্রীয় ধারণা

Image = sequence of patches। Patch = "visual word"। Transformer-এর self-attention all patch-pair relationship modeling — global context থেকেই।

২ · ViT pipeline

  1. Patch: 224×224 image → 14×14 = 196 patch (16×16 each)।
  2. Linear embed: each patch flatten (768-D) → linear → embedding।
  3. Position embed: learnable position vector add।
  4. CLS token: additional learnable token prepend (classification proxy)।
  5. Transformer encoder: 12 layer (ViT-Base) — multi-head attention + FFN।
  6. Classification: CLS token-এর final output → MLP head → class।

৩ · ViT variants

VariantLayersHeadsEmbedParams
ViT-Base121276886M
ViT-Large24161024307M
ViT-Huge32161280632M

৪ · Self-attention quick recap

Each patch query, key, value vector compute। Attention score:

$$\text{Attention}(Q, K, V) = \text{softmax}\left( \frac{Q K^T}{\sqrt{d_k}} \right) V$$

  • $Q$: query — "এই patch কী চাইছে?"
  • $K$: key — "এই patch কী আছে?"
  • $V$: value — "এই patch-এর information"।
  • All-to-all comparison।

৫ · Key result — data hungry

ImageNet (1.4M) trained ViT-Base — ResNet-50-এর চেয়ে worse। কিন্তু:

  • JFT-300M (300M images, Google internal) — ViT outperforms।
  • Reason: CNN-এর inductive bias (locality, translation equivariance) — ViT শিখতে চাই, কিন্তু data-hungry।
  • Sufficient data → ViT learn patterns CNN encode-এ।

৬ · Modern improvements

  • DeiT (২০২১): data-efficient training tricks, ImageNet-only competitive।
  • Swin Transformer (২০২১): windowed local attention, hierarchical।
  • MAE (২০২১): masked autoencoder pretrain — self-supervised।
  • DINOv2 (২০২৩): 1.4B image self-supervised — SOTA features।
  • EVA, BEiT: alternative pretrain strategies।

৭ · Position embedding

Self-attention permutation-invariant — patch order ignore। Position information explicit add:

  • Learnable 1D: simple, original ViT।
  • Sinusoidal 2D: relative position encode।
  • RoPE: rotary position embedding — modern।
CNN = "examine local patches, build up"। ViT = "list all patches, ask each how relates to others"। Different priors, different strengths।
ViT — image as patch sequence Image 224² split 14² flatten ⋮ 196 patches 16² Linear → 768-D + Position + CLS token Transformer Encoder 12 × (MHA + FFN) 86M params MLP head CLS → class No convolution! Pure attention-based Image becomes sequence — NLP-style processing Global attention — every patch knows every other Strong with massive data, struggles with small
ViT — image patch sequence-এ রূপান্তর। Pure transformer encoder, no conv।

৮ · PyTorch-এ ViT

Python · PyTorch 
import torch
from torchvision.models import vit_b_16, ViT_B_16_Weights

w = ViT_B_16_Weights.IMAGENET1K_V1
model = vit_b_16(weights=w)
model.eval()

x = torch.randn(1, 3, 224, 224)
with torch.no_grad():
    out = model(x)
print("Output:", out.shape)             # (1, 1000)
print(f"Params: {sum(p.numel() for p in model.parameters()):,}")  # ~86M

# Inspect patch embedding
print("Conv proj kernel:", model.conv_proj.kernel_size)  # (16, 16)
print("Conv proj stride:", model.conv_proj.stride)        # (16, 16)

    


    

      torchvision ViT — patch projection actually conv 16×16 stride 16 (mathematically equivalent to flatten + linear)। Practical implementation।
    


    
৯ · ViT applications

    
    
      
  • Image classification: ImageNet SOTA (with sufficient pretrain)।
    • 
      
  • CLIP: ViT image encoder, contrastive language alignment।
    • 
      
  • SAM: ViT-H image encoder।
    • 
      
  • DINOv2: self-supervised foundation।
    • 
      
  • Medical imaging: ViT chest X-ray, retinal।
    • 
      
  • Multimodal: ViT + LLM (LLaVA, BLIP)।
    • 
    


    
১০ · Pros & cons summary

    
Pros:

    
    
      
  • Long-range attention native।
    • 
      
  • Scales with data (better than CNN at large data)।
    • 
      
  • NLP architecture share — multimodal easy।
    • 
      
  • Self-supervised pretrain (MAE, DINO) very effective।
    • 
    

    
Cons:

    
    
      
  • Data-hungry — small data CNN better।
    • 
      
  • Quadratic compute in patch count।
    • 
      
  • Mobile deploy hard।
    • 
      
  • Less interpretable than CNN।
    • 
    


    

      ViT modern foundation model-এর backbone। কিন্তু practical small-data deployment-এ ResNet/EfficientNet এখনো dominant। Use case based choose।
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      
        প্র ০১
        ViT-এর "patch as word" — patch size 16 কেন? 8 বা 32 কী effect?
      
      

        
Patch size — ViT-এর primary hyperparameter।

        
16×16 standard:

        
    
          
  • 224 image → 196 patch (14×14)। Sequence length manageable।
    • 
          
  • Compute O(N²) — N=196 reasonable।
    • 
          
  • Most pretrained available।
    • 
        

        
8×8 (smaller patch):

        
    
          
  • 224 image → 784 patch। Sequence 4x longer।
    • 
          
  • Compute 16x — slow।
    • 
          
  • Better fine-grained detection।
    • 
          
  • Used in some medical, satellite।
    • 
        

        
32×32 (larger patch):

        
    
          
  • 224 image → 49 patch. Sequence short।
    • 
          
  • Compute 4x faster।
    • 
          
  • Coarser features।
    • 
          
  • Less common, but used for very large image।
    • 
        

        
Trade-off:

        
    
          
  • Smaller patch = finer detail, more compute।
    • 
          
  • Larger patch = coarser, faster।
    • 
          
  • Sweet spot 16 — empirically validated।
    • 
        

        
Variable patch:

        
    
          
  • Swin — hierarchical, gradual merge।
    • 
          
  • NaViT — variable size, accept any aspect।
    • 
        

        
মূল উপলব্ধি: Patch size — granularity vs compute। Task-specific tune। 16 reasonable default।

      

    


    

      
        প্র ০২
        ViT inductive bias কম — তাই data hungry। Inductive bias কী, CNN-এ কী আছে যা ViT-এ নেই?
      
      

        
Inductive bias = "model's prior assumptions about data structure"। CV-তে critical concept।

        
CNN inductive bias:

        
    
          
  • Locality: nearby pixel related, distant less।
    • 
          
  • Translation equivariance: object position-independent feature।
    • 
          
  • Hierarchy: low-level → high-level structure।
    • 
          
  • Weight sharing: same kernel everywhere।
    • 
        

        
ViT lack:

        
    
          
  • No locality assumption — every patch can attend any।
    • 
          
  • Translation equivariance — only via position embedding (data-dependent)।
    • 
          
  • Hierarchy — flat (all layer same resolution)।
    • 
          
  • No weight sharing across spatial।
    • 
        

        
Effect with small data:

        
    
          
  • CNN — bias correct mostly, data refines।
    • 
          
  • ViT — bias must be learned from data।
    • 
          
  • 1M image — CNN already good, ViT bad।
    • 
          
  • 100M image — ViT learns bias + more, surpasses CNN।
    • 
        

        
Bias-variance trade-off:

        
    
          
  • Strong bias — biased but consistent (CNN)।
    • 
          
  • Weak bias — flexible but data-hungry (ViT)।
    • 
          
  • Right bias-data match key।
    • 
        

        
Hybrid attempts:

        
    
          
  • Swin — local attention (locality bias)।
    • 
          
  • CoAtNet — conv early + attention late।
    • 
          
  • Best of both world।
    • 
        

        
Position embedding role:

        
    
          
  • 2D-aware position help inductive bias।
    • 
          
  • Random init — completely flexible।
    • 
          
  • Sinusoidal — some bias।
    • 
        

        
মূল উপলব্ধি: "No free lunch"। Bias trade-off। Modern direction — task-appropriate bias inject।

      

    


    

      
        প্র ০৩
        MAE (Masked Autoencoder) ViT pretrain — কীভাবে কাজ করে? ImageNet alone ViT competitive কেন এটি দিয়ে?
      
      

        
MAE (He et al., 2021) — ViT-এর "BERT moment"। Self-supervised approach revolutionary।

        
Idea:

        
    
          
  • 75% patch random mask।
    • 
          
  • Encoder process visible 25% only।
    • 
          
  • Decoder reconstruct full image।
    • 
          
  • Loss: MSE reconstruct vs original।
    • 
        

        
Why effective:

        
    
          
  • "Holistic understanding" force — guess hidden from visible।
    • 
          
  • Strong pretext task — generic feature emerge।
    • 
          
  • Asymmetric design — encoder light (75% miss), decoder heavy।
    • 
          
  • Compute-efficient pretrain।
    • 
        

        
NLP parallel:

        
    
          
  • BERT — masked language modeling। Word predict।
    • 
          
  • MAE — masked image modeling। Patch predict।
    • 
          
  • Same philosophy, different domain।
    • 
        

        
Key innovations:

        
    
          
  • High mask ratio (75%): NLP 15%, vision-এ much higher।
    • 
          
  • Asymmetric encoder-decoder: efficient।
    • 
          
  • Light decoder discarded post-pretrain: only encoder for downstream।
    • 
        

        
ImageNet result:

        
    
          
  • ViT-Large from scratch ImageNet: 76%।
    • 
          
  • ViT-Large MAE pretrain + ImageNet finetune: 85.9%।
    • 
          
  • +10% — massive improvement।
    • 
        

        
Compared to ImageNet supervised pretrain:

        
    
          
  • MAE — slightly better।
    • 
          
  • Plus: no label needed for pretrain।
    • 
          
  • Scales with unlabeled data।
    • 
        

        
Subsequent extensions:

        
    
          
  • VideoMAE: spatiotemporal mask।
    • 
          
  • MAE-3D: volumetric medical।
    • 
          
  • ConvNeXt-V2: MAE-style for CNN।
    • 
          
  • EVA, BEiT-2: related approaches।
    • 
        

        
DINOv2:

        
    
          
  • Different self-supervised — distillation-based।
    • 
          
  • 1.4B image, ViT-G (1B params)।
    • 
          
  • Linear probe ImageNet 86.5%।
    • 
          
  • Possibly best vision foundation 2023-2024।
    • 
        

        
মূল উপলব্ধি: Self-supervised — vision-এ NLP-র parallel। Pretraining recipe = key to ViT-এর dominance। Bangladesh AI — pretrained foundation use, finetune local।

      

    


    

      
        প্র ০৪
        ViT mobile-এ inference difficult। Mobile ViT, EfficientFormer — কীভাবে adapt?
      
      

        
Mobile vision — ViT-এর Achilles heel। Active research area।

        
Why ViT mobile-unfriendly:

        
    
          
  • Self-attention $O(N^2)$ — sequence length-এ quadratic।
    • 
          
  • 196 patch × 196 attention matrix = 38K compute per layer।
    • 
          
  • Mobile NPU ARM-optimized for convolution।
    • 
          
  • Memory bandwidth bottleneck।
    • 
        

        
MobileViT (২০২২, Apple):

        
    
          
  • Hybrid: conv early + transformer block intermediate।
    • 
          
  • Local attention windows — compute manageable।
    • 
          
  • 5M params, similar to MobileNetV3।
    • 
          
  • Performance — better than MobileNet।
    • 
        

        
EfficientFormer (২০২২):

        
    
          
  • Hybrid: 4D conv blocks early, transformer late।
    • 
          
  • Small-resolution attention (downsampled)।
    • 
          
  • 2 ms inference iPhone (real-time)।
    • 
        

        
FastViT (২০২৩, Apple):

        
    
          
  • Replace attention with structural reparameterization।
    • 
          
  • Conv at training, simplify at inference।
    • 
          
  • iPhone real-time।
    • 
        

        
NextViT (২০২২):

        
    
          
  • Convolution + transformer hybrid block।
    • 
          
  • Industry application focus।
    • 
        

        
EdgeNeXt:

        
    
          
  • Split-depth-wise transpose attention।
    • 
          
  • Mobile-friendly variations।
    • 
        

        
Common patterns:

        
    
          
  • Conv early (locality)।
    • 
          
  • Attention late (semantic)।
    • 
          
  • Window-based attention।
    • 
          
  • Reduced attention head count।
    • 
          
  • Linear/efficient attention।
    • 
        

        
Bangladesh deployment:

        
    
          
  • Low-end Android — pure CNN (MobileNetV3) safer।
    • 
          
  • Mid-range — MobileViT, EfficientFormer।
    • 
          
  • iPhone — FastViT iOS-optimize।
    • 
        

        
Quantization:

        
    
          
  • ViT FP32 → INT8 — typically 2-3% accuracy drop।
    • 
          
  • 4-bit possible with care।
    • 
          
  • Distillation help।
    • 
        

        
মূল উপলব্ধি: Pure ViT mobile-এ rare এখনো। Hybrid future। Bangladesh-এর diverse mobile market — multi-architecture pipeline।

      

    

  


  

    
 অনুশীলন

    
    
      
  1. 
        Patch count: 384×384 image, patch 16 — sequence length?
        
    
          
          
    
            
    $(384/16)^2 + 1 = 24^2 + 1 = 577$ (CLS token সহ)।
    
          
    
        
    
      
    2. 

      
  2. 
        ViT vs ResNet: Same task, same data — কোনটা bigger likely better?
        
    
          
          
    
            
    1K image — ResNet। 100K image — close। 10M image — ViT। Pretrained ViT (DINOv2) finetune almost always wins।
    
          
    
        
    
      
    3. 

      
  3. 
        ভাবুন: Bangla street sign — 200 image। ViT-Base scratch fail করবে — কী fix?
        
    
          
          
    
            
    (1) DINOv2 pretrained backbone use। (2) Linear probe বা LoRA finetune। (3) Heavy augment। (4) Alternatively MobileNetV3 simpler but more reliable on small data।
    
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন · ABCL TECH-এ আপনার পরবর্তী পদক্ষেপ

    
    

      কোড রানার কাজ না করলে? Google Colab use করুন।
    

  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ২৪ · SAM