পাঠ ২৫ · ৩৫-এর মধ্যে · মডিউল ৪
Home / AI Courses / Computer Vision / ViT

Vision Transformer (ViT)

Vision Transformer — image as sequence of patches
৮ মিনিট পড়া উচ্চ · Advanced PyTorch কোডসহ

এই পাঠে যা শিখবেন

  • ViT architecture step-by-step
  • Patch embedding, position embedding
  • Self-attention CV-তে
  • ViT vs CNN trade-off

১ · ViT-এর big idea

Transformer (২০১৭) NLP-এ revolution. ২০২০ পর্যন্ত CV-তে CNN dominant. Dosovitskiy et al. — "We do not need convolution at all".

Approach: image-কে non-overlapping patches-এ ভাগ → প্রতিটি patch = "word". Standard NLP transformer apply.

কেন্দ্রীয় ধারণা

Image = sequence of patches. Patch = "visual word". Transformer-এর self-attention all patch-pair relationship modeling — global context থেকেই।

২ · ViT pipeline

  1. Patch: 224×224 image → 14×14 = 196 patch (16×16 each).
  2. Linear embed: each patch flatten (768-D) → linear → embedding.
  3. Position embed: learnable position vector add.
  4. CLS token: additional learnable token prepend (classification proxy).
  5. Transformer encoder: 12 layer (ViT-Base) — multi-head attention + FFN.
  6. Classification: CLS token-এর final output → MLP head → class.

৩ · ViT variants

VariantLayersHeadsEmbedParams
ViT-Base121276886M
ViT-Large24161024307M
ViT-Huge32161280632M

৪ · Self-attention quick recap

Each patch query, key, value vector compute. Attention score:

$$\text{Attention}(Q, K, V) = \text{softmax}\left( \frac{Q K^T}{\sqrt{d_k}} \right) V$$

  • $Q$: query — "এই patch কী চাইছে?"
  • $K$: key — "এই patch কী আছে?"
  • $V$: value — "এই patch-এর information".
  • All-to-all comparison.

৫ · Key result — data hungry

ImageNet (1.4M) trained ViT-Base — ResNet-50-এর চেয়ে worse. কিন্তু:

  • JFT-300M (300M images, Google internal) — ViT outperforms.
  • Reason: CNN-এর inductive bias (locality, translation equivariance) — ViT শিখতে চাই, কিন্তু data-hungry.
  • Sufficient data → ViT learn patterns CNN encode-এ।

৬ · Modern improvements

  • DeiT (২০২১): data-efficient training tricks, ImageNet-only competitive.
  • Swin Transformer (২০২১): windowed local attention, hierarchical.
  • MAE (২০২১): masked autoencoder pretrain — self-supervised.
  • DINOv2 (২০২৩): 1.4B image self-supervised — SOTA features.
  • EVA, BEiT: alternative pretrain strategies.

৭ · Position embedding

Self-attention permutation-invariant — patch order ignore. Position information explicit add:

  • Learnable 1D: simple, original ViT.
  • Sinusoidal 2D: relative position encode.
  • RoPE: rotary position embedding — modern.
CNN = "examine local patches, build up". ViT = "list all patches, ask each how relates to others". Different priors, different strengths.
ViT — image as patch sequence Image 224² split 14² flatten ⋮ 196 patches 16² Linear → 768-D + Position + CLS token Transformer Encoder 12 × (MHA + FFN) 86M params MLP head CLS → class No convolution! Pure attention-based Image becomes sequence — NLP-style processing Global attention — every patch knows every other Strong with massive data, struggles with small
ViT — image patch sequence-এ রূপান্তর। Pure transformer encoder, no conv.

৮ · PyTorch-এ ViT

Python · PyTorch 
import torch
from torchvision.models import vit_b_16, ViT_B_16_Weights

w = ViT_B_16_Weights.IMAGENET1K_V1
model = vit_b_16(weights=w)
model.eval()

x = torch.randn(1, 3, 224, 224)
with torch.no_grad():
    out = model(x)
print("Output:", out.shape)             # (1, 1000)
print(f"Params: {sum(p.numel() for p in model.parameters()):,}")  # ~86M

# Inspect patch embedding
print("Conv proj kernel:", model.conv_proj.kernel_size)  # (16, 16)
print("Conv proj stride:", model.conv_proj.stride)        # (16, 16)

    


    

      torchvision ViT — patch projection actually conv 16×16 stride 16 (mathematically equivalent to flatten + linear). Practical implementation.
    


    
৯ · ViT applications

    
    
      
  • Image classification: ImageNet SOTA (with sufficient pretrain).
    • 
      
  • CLIP: ViT image encoder, contrastive language alignment.
    • 
      
  • SAM: ViT-H image encoder.
    • 
      
  • DINOv2: self-supervised foundation.
    • 
      
  • Medical imaging: ViT chest X-ray, retinal.
    • 
      
  • Multimodal: ViT + LLM (LLaVA, BLIP).
    • 
    


    
১০ · Pros & cons summary

    
Pros:

    
    
      
  • Long-range attention native.
    • 
      
  • Scales with data (better than CNN at large data).
    • 
      
  • NLP architecture share — multimodal easy.
    • 
      
  • Self-supervised pretrain (MAE, DINO) very effective.
    • 
    

    
Cons:

    
    
      
  • Data-hungry — small data CNN better.
    • 
      
  • Quadratic compute in patch count.
    • 
      
  • Mobile deploy hard.
    • 
      
  • Less interpretable than CNN.
    • 
    


    

      ViT modern foundation model-এর backbone. কিন্তু practical small-data deployment-এ ResNet/EfficientNet এখনো dominant. Use case based choose.
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      
        প্র ০১
        ViT-এর "patch as word" — patch size 16 কেন? 8 বা 32 কী effect?
      
      

        
Patch size — ViT-এর primary hyperparameter.

        
16×16 standard:

        
    
          
  • 224 image → 196 patch (14×14). Sequence length manageable.
    • 
          
  • Compute O(N²) — N=196 reasonable.
    • 
          
  • Most pretrained available.
    • 
        

        
8×8 (smaller patch):

        
    
          
  • 224 image → 784 patch. Sequence 4x longer.
    • 
          
  • Compute 16x — slow.
    • 
          
  • Better fine-grained detection.
    • 
          
  • Used in some medical, satellite.
    • 
        

        
32×32 (larger patch):

        
    
          
  • 224 image → 49 patch. Sequence short.
    • 
          
  • Compute 4x faster.
    • 
          
  • Coarser features.
    • 
          
  • Less common, but used for very large image.
    • 
        

        
Trade-off:

        
    
          
  • Smaller patch = finer detail, more compute.
    • 
          
  • Larger patch = coarser, faster.
    • 
          
  • Sweet spot 16 — empirically validated.
    • 
        

        
Variable patch:

        
    
          
  • Swin — hierarchical, gradual merge.
    • 
          
  • NaViT — variable size, accept any aspect.
    • 
        

        
মূল উপলব্ধি: Patch size — granularity vs compute. Task-specific tune. 16 reasonable default.

      

    


    

      
        প্র ০২
        ViT inductive bias কম — তাই data hungry. Inductive bias কী, CNN-এ কী আছে যা ViT-এ নেই?
      
      

        
Inductive bias = "model's prior assumptions about data structure". CV-তে critical concept.

        
CNN inductive bias:

        
    
          
  • Locality: nearby pixel related, distant less.
    • 
          
  • Translation equivariance: object position-independent feature.
    • 
          
  • Hierarchy: low-level → high-level structure.
    • 
          
  • Weight sharing: same kernel everywhere.
    • 
        

        
ViT lack:

        
    
          
  • No locality assumption — every patch can attend any.
    • 
          
  • Translation equivariance — only via position embedding (data-dependent).
    • 
          
  • Hierarchy — flat (all layer same resolution).
    • 
          
  • No weight sharing across spatial.
    • 
        

        
Effect with small data:

        
    
          
  • CNN — bias correct mostly, data refines.
    • 
          
  • ViT — bias must be learned from data.
    • 
          
  • 1M image — CNN already good, ViT bad.
    • 
          
  • 100M image — ViT learns bias + more, surpasses CNN.
    • 
        

        
Bias-variance trade-off:

        
    
          
  • Strong bias — biased but consistent (CNN).
    • 
          
  • Weak bias — flexible but data-hungry (ViT).
    • 
          
  • Right bias-data match key.
    • 
        

        
Hybrid attempts:

        
    
          
  • Swin — local attention (locality bias).
    • 
          
  • CoAtNet — conv early + attention late.
    • 
          
  • Best of both world.
    • 
        

        
Position embedding role:

        
    
          
  • 2D-aware position help inductive bias.
    • 
          
  • Random init — completely flexible.
    • 
          
  • Sinusoidal — some bias.
    • 
        

        
মূল উপলব্ধি: "No free lunch". Bias trade-off. Modern direction — task-appropriate bias inject.

      

    


    

      
        প্র ০৩
        MAE (Masked Autoencoder) ViT pretrain — কীভাবে কাজ করে? ImageNet alone ViT competitive কেন এটি দিয়ে?
      
      

        
MAE (He et al., 2021) — ViT-এর "BERT moment". Self-supervised approach revolutionary.

        
Idea:

        
    
          
  • 75% patch random mask.
    • 
          
  • Encoder process visible 25% only.
    • 
          
  • Decoder reconstruct full image.
    • 
          
  • Loss: MSE reconstruct vs original.
    • 
        

        
Why effective:

        
    
          
  • "Holistic understanding" force — guess hidden from visible.
    • 
          
  • Strong pretext task — generic feature emerge.
    • 
          
  • Asymmetric design — encoder light (75% miss), decoder heavy.
    • 
          
  • Compute-efficient pretrain.
    • 
        

        
NLP parallel:

        
    
          
  • BERT — masked language modeling. Word predict.
    • 
          
  • MAE — masked image modeling. Patch predict.
    • 
          
  • Same philosophy, different domain.
    • 
        

        
Key innovations:

        
    
          
  • High mask ratio (75%): NLP 15%, vision-এ much higher.
    • 
          
  • Asymmetric encoder-decoder: efficient.
    • 
          
  • Light decoder discarded post-pretrain: only encoder for downstream.
    • 
        

        
ImageNet result:

        
    
          
  • ViT-Large from scratch ImageNet: 76%.
    • 
          
  • ViT-Large MAE pretrain + ImageNet finetune: 85.9%.
    • 
          
  • +10% — massive improvement.
    • 
        

        
Compared to ImageNet supervised pretrain:

        
    
          
  • MAE — slightly better.
    • 
          
  • Plus: no label needed for pretrain.
    • 
          
  • Scales with unlabeled data.
    • 
        

        
Subsequent extensions:

        
    
          
  • VideoMAE: spatiotemporal mask.
    • 
          
  • MAE-3D: volumetric medical.
    • 
          
  • ConvNeXt-V2: MAE-style for CNN.
    • 
          
  • EVA, BEiT-2: related approaches.
    • 
        

        
DINOv2:

        
    
          
  • Different self-supervised — distillation-based.
    • 
          
  • 1.4B image, ViT-G (1B params).
    • 
          
  • Linear probe ImageNet 86.5%.
    • 
          
  • Possibly best vision foundation 2023-2024.
    • 
        

        
মূল উপলব্ধি: Self-supervised — vision-এ NLP-র parallel. Pretraining recipe = key to ViT-এর dominance. Bangladesh AI — pretrained foundation use, finetune local.

      

    


    

      
        প্র ০৪
        ViT mobile-এ inference difficult. Mobile ViT, EfficientFormer — কীভাবে adapt?
      
      

        
Mobile vision — ViT-এর Achilles heel. Active research area.

        
Why ViT mobile-unfriendly:

        
    
          
  • Self-attention $O(N^2)$ — sequence length-এ quadratic.
    • 
          
  • 196 patch × 196 attention matrix = 38K compute per layer.
    • 
          
  • Mobile NPU ARM-optimized for convolution.
    • 
          
  • Memory bandwidth bottleneck.
    • 
        

        
MobileViT (২০২২, Apple):

        
    
          
  • Hybrid: conv early + transformer block intermediate.
    • 
          
  • Local attention windows — compute manageable.
    • 
          
  • 5M params, similar to MobileNetV3.
    • 
          
  • Performance — better than MobileNet.
    • 
        

        
EfficientFormer (২০২২):

        
    
          
  • Hybrid: 4D conv blocks early, transformer late.
    • 
          
  • Small-resolution attention (downsampled).
    • 
          
  • 2 ms inference iPhone (real-time).
    • 
        

        
FastViT (২০২৩, Apple):

        
    
          
  • Replace attention with structural reparameterization.
    • 
          
  • Conv at training, simplify at inference.
    • 
          
  • iPhone real-time.
    • 
        

        
NextViT (২০২২):

        
    
          
  • Convolution + transformer hybrid block.
    • 
          
  • Industry application focus.
    • 
        

        
EdgeNeXt:

        
    
          
  • Split-depth-wise transpose attention.
    • 
          
  • Mobile-friendly variations.
    • 
        

        
Common patterns:

        
    
          
  • Conv early (locality).
    • 
          
  • Attention late (semantic).
    • 
          
  • Window-based attention.
    • 
          
  • Reduced attention head count.
    • 
          
  • Linear/efficient attention.
    • 
        

        
Bangladesh deployment:

        
    
          
  • Low-end Android — pure CNN (MobileNetV3) safer.
    • 
          
  • Mid-range — MobileViT, EfficientFormer.
    • 
          
  • iPhone — FastViT iOS-optimize.
    • 
        

        
Quantization:

        
    
          
  • ViT FP32 → INT8 — typically 2-3% accuracy drop.
    • 
          
  • 4-bit possible with care.
    • 
          
  • Distillation help.
    • 
        

        
মূল উপলব্ধি: Pure ViT mobile-এ rare এখনো। Hybrid future. Bangladesh-এর diverse mobile market — multi-architecture pipeline.

      

    

  


  

    
 অনুশীলন

    
    
      
  1. 
        Patch count: 384×384 image, patch 16 — sequence length?
        
    
          
          
    
            
    $(384/16)^2 + 1 = 24^2 + 1 = 577$ (CLS token সহ)।
    
          
    
        
    
      
    2. 

      
  2. 
        ViT vs ResNet: Same task, same data — কোনটা bigger likely better?
        
    
          
          
    
            
    1K image — ResNet. 100K image — close. 10M image — ViT. Pretrained ViT (DINOv2) finetune almost always wins.
    
          
    
        
    
      
    3. 

      
  3. 
        ভাবুন: Bangla street sign — 200 image. ViT-Base scratch fail করবে — কী fix?
        
    
          
          
    
            
    (1) DINOv2 pretrained backbone use. (2) Linear probe বা LoRA finetune. (3) Heavy augment. (4) Alternatively MobileNetV3 simpler but more reliable on small data.
    
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন · ABCL TECH-এ আপনার পরবর্তী পদক্ষেপ

    
    

      কোড রানার কাজ না করলে? Google Colab use করুন।
    

  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ২৪ · SAM