পাঠ ২৬ · ৩৫-এর মধ্যে · মডিউল ৪
Home / AI Courses / Computer Vision / Swin

Swin Transformer

Swin Transformer — shifted window attention
৭ মিনিট পড়া উচ্চ · Advanced PyTorch কোডসহ

এই পাঠে যা শিখবেন

  • Window-based attention
  • Shifted window — cross-window connection
  • Hierarchical feature build-up
  • Detection/segmentation backbone use

১ · ViT-এর সমস্যা detection-এ

Vanilla ViT — same resolution all layer। FPN-style multi-scale support নেই। Detection-এ সমস্যা।

Compute issue: 1024×1024 image, 16 patch → 4096 token। Self-attention $O(N^2)$ — 16M operation।

কেন্দ্রীয় ধারণা

Swin = ViT + locality + hierarchy। Window-attention compute reduce, shifted window-এ cross-window connection, stage-wise downsample CNN-like।

২ · Window attention

Image-কে non-overlapping windows-এ ভাগ (e.g., 7×7 patch per window)। Attention শুধু window-এর ভিতরে compute।

  • Standard attention: $O(N^2)$।
  • Window attention: $O(N \cdot M^2)$ যেখানে $M$ = window size।
  • $M$ fixed → linear in $N$।

৩ · Shifted window — connect across

Pure window-attention isolated — windows কখনো communicate করে না।

Shifted window: alternate layer-এ window shift (M/2)। নতুন windows previous layers-এর content cross করে।

  • Layer 1: regular window।
  • Layer 2: shifted window — half-overlap with layer 1।
  • Two layer-এর combination — full coverage।

৪ · Hierarchical structure

4 stage, each stage:

  • Patch merge (2×2 → 1) — spatial downsample 2x।
  • Channel double।
  • Multiple Swin block।

Stage progression: $H/4 \times W/4 \times 96 \to H/8 \times W/8 \times 192 \to H/16 \times W/16 \times 384 \to H/32 \times W/32 \times 768$।

ResNet-এর মতো — multi-scale feature naturally available। Detection FPN ready।

৫ · Swin variants

  • Swin-T (Tiny): 28M params।
  • Swin-S (Small): 50M।
  • Swin-B (Base): 88M।
  • Swin-L (Large): 197M।

৬ · Swin v2 (২০২২)

  • 3 billion parameter version।
  • Larger window (24 vs 7)।
  • Cosine attention — stability।
  • Log-spaced relative position bias।
  • Higher resolution support।
ViT = "all-to-all chat" — global but expensive। Swin = "small group chats with reshuffling" — local efficient + cross-group sharing। Best of both — locality + global coverage।
Swin Transformer — windowed hierarchical Stage 1 H/4 × W/4 × 96 2 Swin blocks window 7×7 Stage 2 H/8 × W/8 × 192 2 Swin blocks Stage 3 H/16² × 384, 6 blocks Stage 4 H/32²×768 Shifted window mechanism Layer L regular window Layer L+1 shifted window → cross-window info flow Hierarchy + windowed attention = efficient + multi-scale
Swin — 4 stage hierarchical, alternating regular/shifted windows। Detection-friendly multi-scale।

৭ · PyTorch use

Python · PyTorch 
import torch
from torchvision.models import swin_t, Swin_T_Weights

w = Swin_T_Weights.IMAGENET1K_V1
model = swin_t(weights=w)
model.eval()

x = torch.randn(1, 3, 224, 224)
with torch.no_grad():
    out = model(x)
print("Output:", out.shape)             # (1, 1000)
print(f"Params: {sum(p.numel() for p in model.parameters()):,}")  # ~28M

# Multi-scale features (manually extract)
features = []
def hook(m, i, o):
    features.append(o)
# Hook stage outputs for FPN-style use

    


    

      Swin-T 28M params — ResNet-50 (25M) similar size, kintu often higher accuracy। Modern detection backbone preferred।
    


    
৮ · Use cases

    
    
      
  • Object detection: Swin + Mask R-CNN — COCO SOTA।
    • 
      
  • Semantic segmentation: Swin + UperNet — ADE20K SOTA।
    • 
      
  • Instance segmentation: Mask2Former + Swin।
    • 
      
  • Video: Video Swin Transformer।
    • 
      
  • Medical: Swin-UNet — 3D segmentation।
    • 
    


    
৯ · ConvNeXt — CNN strikes back

    

      Liu et al. (২০২২) — "A ConvNet for the 2020s"। ResNet-কে modern training trick + ViT design choice দিয়ে modernize। Swin-এর সাথে competitive।
    

    
    
      
  • 7×7 depthwise conv (Swin-এর 7-window equivalent)।
    • 
      
  • LayerNorm, GELU।
    • 
      
  • Inverted bottleneck।
    • 
      
  • Pure CNN, no attention।
    • 
      
  • Often beats Swin on similar setting।
    • 
    


    
১০ · Swin vs ConvNeXt vs ViT

    

      
        
ModelImageNetDetection (COCO)Mobile

      
      
        
ViT-B81.8%N/A directHard

        
Swin-B83.5%51.9Medium

        
ConvNeXt-B83.8%52.6Better

        
EfficientNet-B784.3%N/AYes

      
    


    

      Swin — २०२१-২০২২-এর dominant detection backbone। ২০২৪+ — ConvNeXt-V2, EVA, DINO-v2 backbone competing। Architecture race ongoing।
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      
        প্র ০১
        Window attention compute reduce, কিন্তু long-range dependency limit। Practical impact কী?
      
      

        
Window-attention trade-off — efficiency vs global context।

        
ViT global attention:

        
    
          
  • Layer 1 — every patch sees every patch।
    • 
          
  • Long-range dependency immediate।
    • 
          
  • Compute O(N²)।
    • 
        

        
Swin local attention:

        
    
          
  • Layer 1 — patch sees only window (49 patch)।
    • 
          
  • Long-range dependency build through stages।
    • 
          
  • Stage 4 — receptive field eventually covers everything।
    • 
          
  • Compute linear।
    • 
        

        
Effective receptive field:

        
    
          
  • Stage 1: 7×7 window।
    • 
          
  • Stage 2: shifted + downsample → 14×14 effective।
    • 
          
  • Stage 4: 7×7 window-এ 32x downsample = 224×224 effective।
    • 
          
  • Full image cover by stage 4।
    • 
        

        
Practical impact:

        
    
          
  • Shape, size, fine-grained — Swin equally good।
    • 
          
  • Truly long-range (e.g., scene context cross-half image) — slightly worse than ViT।
    • 
          
  • Most CV task — local sufficient।
    • 
        

        
Mitigation:

        
    
          
  • Larger window (Swin v2 — 24×24)।
    • 
          
  • Global attention layers occasionally।
    • 
          
  • Cross-window mechanisms।
    • 
        

        
Real comparison:

        
    
          
  • ImageNet — Swin within 0.5% of ViT।
    • 
          
  • Detection — Swin BETTER (multi-scale)।
    • 
          
  • Segmentation — Swin BETTER।
    • 
          
  • Long-range reasoning task — ViT slight edge।
    • 
        

        
মূল উপলব্ধি: "Global attention always better" — myth। Most vision task local + hierarchy যথেষ্ট। Swin-এর efficiency win।

      

    


    

      
        প্র ০২
        Shifted window — implementation-এ tricky। Cyclic shift trick কী, padding/masking কীভাবে?
      
      

        
Shifted window-এর efficient implementation — engineering brilliance।

        
Naive shift:

        
    
          
  • Window positions shift M/2।
    • 
          
  • Edge windows partial — pad zero।
    • 
          
  • More windows (e.g., 9 instead of 4)।
    • 
          
  • Compute increase।
    • 
        

        
Cyclic shift trick:

        
    
          
  • Image-কে cyclic shift (top→bottom, left→right wrap)।
    • 
          
  • Windows now regular grid again।
    • 
          
  • Same compute as non-shifted।
    • 
          
  • Reverse shift after attention।
    • 
        

        
Masking:

        
    
          
  • Cyclic shift-এ — cross-image-boundary patches mixed।
    • 
          
  • Attention-এ mask — invalid pair prevent।
    • 
          
  • Mask pre-computed, reuse।
    • 
        

        
Implementation:

        
def shift_window(x, shift):
    # x: (B, H, W, C)
    return torch.roll(x, shifts=(-shift, -shift), dims=(1, 2))

def reverse_shift(x, shift):
    return torch.roll(x, shifts=(shift, shift), dims=(1, 2))

# In Swin block
if shift > 0:
    x = shift_window(x, shift)
    attn = window_attention(x, mask)
    x = reverse_shift(attn, shift)
else:
    x = window_attention(x)

        
Mask construction:

        
    
          
  • Per window — patches from different "regions" identify।
    • 
          
  • Within-region attention OK।
    • 
          
  • Cross-region attention masked।
    • 
          
  • Once compute, reuse for all forward।
    • 
        

        
Performance:

        
    
          
  • Cyclic + mask — same compute as non-shifted।
    • 
          
  • Memory minimal overhead।
    • 
          
  • Backward pass also efficient।
    • 
        

        
Lesson:

        
    
          
  • Algorithm correctness ≠ efficient implementation।
    • 
          
  • Hardware-aware design crucial।
    • 
          
  • Engineering art — research paper-এর underrated component।
    • 
        

        
মূল উপলব্ধি: Implementation cleverness — Swin-এর success-এর key। "Pretty algorithm" alone insufficient — efficient code matters।

      

    


    

      
        প্র ০৩
        Swin detection-এ champion ছিল ২০২১। ২০২৬-এ — DINO-DETR, Mask2Former এ Swin backbone vs full transformer? Trade-off?
      
      

        
Architecture wars-এর latest chapter।

        
Swin backbone:

        
    
          
  • Hierarchical features → FPN compatible।
    • 
          
  • Detection head choice flexible।
    • 
          
  • Training mature, code well-tested।
    • 
          
  • Mask R-CNN, DETR — both works।
    • 
        

        
ViT backbone (vanilla):

        
    
          
  • Single-scale — detection awkward।
    • 
          
  • Modify (e.g., ViTDet) make work।
    • 
          
  • Self-supervised pretrain (MAE) very strong।
    • 
        

        
DINO-DETR + Swin:

        
    
          
  • Swin backbone + DETR head।
    • 
          
  • COCO 63.3 mAP — top performer।
    • 
          
  • Proven combination।
    • 
        

        
EVA-02 + Mask2Former:

        
    
          
  • EVA pretrained ViT।
    • 
          
  • Mask2Former head।
    • 
          
  • Slightly better than Swin equivalent।
    • 
        

        
ConvNeXt + DINO:

        
    
          
  • Pure CNN backbone with modern training।
    • 
          
  • Mobile-friendlier।
    • 
          
  • Similar detection accuracy।
    • 
        

        
Compute trade-off:

        
    
          
  • Swin — moderate compute।
    • 
          
  • ViT-MAE — slightly more।
    • 
          
  • EVA — heavy।
    • 
          
  • ConvNeXt — efficient।
    • 
        

        
Production recommendation 2026:

        
    
          
  • SOTA accuracy: EVA-02 + Mask2Former।
    • 
          
  • Best speed/accuracy: Swin-B + DINO-DETR।
    • 
          
  • Mobile: ConvNeXt-Tiny + YOLO।
    • 
          
  • Realtime: RT-DETR (Swin/ResNet variants)।
    • 
        

        
Bangladesh practical:

        
    
          
  • GPU-rich research — EVA-02।
    • 
          
  • Production server — Swin-B।
    • 
          
  • Edge — ConvNeXt-Tiny।
    • 
        

        
মূল উপলব্ধি: "Best architecture" task + compute + ecosystem-dependent। Swin still strong, ViT/CNN/hybrid-এর mix — modern reality।

      

    


    

      
        প্র ০৪
        Swin v2 (২০২২) 3B params — vision foundation model। Trillion-parameter vision আছে কি? Future direction?
      
      

        
Vision scaling — NLP-এর pace follow করছে।

        
Vision model size timeline:

        
    
          
  • ২০১২ AlexNet: 60M।
    • 
          
  • ২০১৭ ResNeXt-101: 84M।
    • 
          
  • ২০১৯ EfficientNet-B7: 66M।
    • 
          
  • ২০২০ ViT-Huge: 632M।
    • 
          
  • ২০২২ Swin-V2-3B: 3B।
    • 
          
  • ২০২২-২০২৩ ViT-22B (Google): 22B।
    • 
          
  • ২০২৩ EVA-02: 1B।
    • 
          
  • ২০২৪ DINOv2: ~1B।
    • 
        

        
Compared to LLM:

        
    
          
  • GPT-3: 175B।
    • 
          
  • GPT-4: speculatively 1T+।
    • 
          
  • Llama-3 405B।
    • 
          
  • Vision lagging by 100x।
    • 
        

        
Why slower scaling vision:

        
    
          
  • Image data scarcer than text (curated)।
    • 
          
  • Cost — each image 100x text-bytes।
    • 
          
  • Pretraining objective less clear (MAE vs DINO vs supervised)।
    • 
          
  • Less commercial pressure।
    • 
        

        
Multimodal era:

        
    
          
  • Vision + language combine — GPT-4V, Gemini।
    • 
          
  • Vision encoder typically smaller (1-10B)।
    • 
          
  • Most "intelligence" in language part।
    • 
        

        
Where pure vision scale:

        
    
          
  • Medical imaging — DINOv2 + finetune।
    • 
          
  • Satellite/remote sensing — billion-scale।
    • 
          
  • Industrial — task-specific large model।
    • 
        

        
Future trends:

        
    
          
  • Multimodal native (image + video + text + audio)।
    • 
          
  • Vision-language pretraining dominant।
    • 
          
  • Pure vision foundation model role narrow but persistent।
    • 
        

        
Bangladesh implication:

        
    
          
  • Train such model impossible locally।
    • 
          
  • Foundation model API (Anthropic, OpenAI) leverage।
    • 
          
  • Local fine-tune (LoRA) only realistic option।
    • 
          
  • Open foundation model (DINOv2, Llama) preferred।
    • 
        

        
মূল উপলব্ধি: Vision foundation model age dawning। Bangladesh — adapter, applier, evaluator। Research-funded scaling unattainable, application-layer rich।

      

    

  


  

    
 অনুশীলন

    
    
      
  1. 
        Compute compare: 224×224 image, patch 4 (Swin-T) → token count? Window 7×7 — windows কত?
        
    
          
          
    
            
    Token: $(224/4)^2 = 56^2 = 3136$। Window: $56/7 = 8$ per row, $8^2 = 64$ windows।
    
          
    
        
    
      
    2. 

      
  2. 
        Pretrained: torchvision Swin-T load and inference।
        
    
          
          
    
            
    from torchvision.models import swin_t
m = swin_t(weights='DEFAULT').eval()
out = m(torch.randn(1, 3, 224, 224))
    
          
    
        
    
      
    3. 

      
  3. 
        ভাবুন: Bangladesh medical 3D MRI segment — Swin-UNet vs U-Net কখন কোনটা?
        
    
          
          
    
            
    Small dataset (under 100 volumes) → U-Net। Large multi-hospital dataset (1000+) → Swin-UNet — long-range context capture better।
    
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন · ABCL TECH-এ আপনার পরবর্তী পদক্ষেপ

    
    

      কোড রানার কাজ না করলে? Google Colab use করুন।
    

  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ২৫ · Vision Transformer