পাঠ ২৬ · ৩৫-এর মধ্যে · মডিউল ৪
Home / AI Courses / Computer Vision / Swin

Swin Transformer

Swin Transformer — shifted window attention
৭ মিনিট পড়া উচ্চ · Advanced PyTorch কোডসহ

এই পাঠে যা শিখবেন

  • Window-based attention
  • Shifted window — cross-window connection
  • Hierarchical feature build-up
  • Detection/segmentation backbone use

১ · ViT-এর সমস্যা detection-এ

Vanilla ViT — same resolution all layer. FPN-style multi-scale support নেই। Detection-এ সমস্যা।

Compute issue: 1024×1024 image, 16 patch → 4096 token. Self-attention $O(N^2)$ — 16M operation.

কেন্দ্রীয় ধারণা

Swin = ViT + locality + hierarchy. Window-attention compute reduce, shifted window-এ cross-window connection, stage-wise downsample CNN-like.

২ · Window attention

Image-কে non-overlapping windows-এ ভাগ (e.g., 7×7 patch per window). Attention শুধু window-এর ভিতরে compute.

  • Standard attention: $O(N^2)$।
  • Window attention: $O(N \cdot M^2)$ যেখানে $M$ = window size.
  • $M$ fixed → linear in $N$।

৩ · Shifted window — connect across

Pure window-attention isolated — windows কখনো communicate করে না।

Shifted window: alternate layer-এ window shift (M/2). নতুন windows previous layers-এর content cross করে।

  • Layer 1: regular window.
  • Layer 2: shifted window — half-overlap with layer 1.
  • Two layer-এর combination — full coverage.

৪ · Hierarchical structure

4 stage, each stage:

  • Patch merge (2×2 → 1) — spatial downsample 2x.
  • Channel double.
  • Multiple Swin block.

Stage progression: $H/4 \times W/4 \times 96 \to H/8 \times W/8 \times 192 \to H/16 \times W/16 \times 384 \to H/32 \times W/32 \times 768$।

ResNet-এর মতো — multi-scale feature naturally available. Detection FPN ready.

৫ · Swin variants

  • Swin-T (Tiny): 28M params.
  • Swin-S (Small): 50M.
  • Swin-B (Base): 88M.
  • Swin-L (Large): 197M.

৬ · Swin v2 (২০২২)

  • 3 billion parameter version.
  • Larger window (24 vs 7).
  • Cosine attention — stability.
  • Log-spaced relative position bias.
  • Higher resolution support.
ViT = "all-to-all chat" — global but expensive. Swin = "small group chats with reshuffling" — local efficient + cross-group sharing. Best of both — locality + global coverage.
Swin Transformer — windowed hierarchical Stage 1 H/4 × W/4 × 96 2 Swin blocks window 7×7 Stage 2 H/8 × W/8 × 192 2 Swin blocks Stage 3 H/16² × 384, 6 blocks Stage 4 H/32²×768 Shifted window mechanism Layer L regular window Layer L+1 shifted window → cross-window info flow Hierarchy + windowed attention = efficient + multi-scale
Swin — 4 stage hierarchical, alternating regular/shifted windows. Detection-friendly multi-scale.

৭ · PyTorch use

Python · PyTorch 
import torch
from torchvision.models import swin_t, Swin_T_Weights

w = Swin_T_Weights.IMAGENET1K_V1
model = swin_t(weights=w)
model.eval()

x = torch.randn(1, 3, 224, 224)
with torch.no_grad():
    out = model(x)
print("Output:", out.shape)             # (1, 1000)
print(f"Params: {sum(p.numel() for p in model.parameters()):,}")  # ~28M

# Multi-scale features (manually extract)
features = []
def hook(m, i, o):
    features.append(o)
# Hook stage outputs for FPN-style use

    


    

      Swin-T 28M params — ResNet-50 (25M) similar size, kintu often higher accuracy. Modern detection backbone preferred.
    


    
৮ · Use cases

    
    
      
  • Object detection: Swin + Mask R-CNN — COCO SOTA.
    • 
      
  • Semantic segmentation: Swin + UperNet — ADE20K SOTA.
    • 
      
  • Instance segmentation: Mask2Former + Swin.
    • 
      
  • Video: Video Swin Transformer.
    • 
      
  • Medical: Swin-UNet — 3D segmentation.
    • 
    


    
৯ · ConvNeXt — CNN strikes back

    

      Liu et al. (২০২২) — "A ConvNet for the 2020s". ResNet-কে modern training trick + ViT design choice দিয়ে modernize. Swin-এর সাথে competitive.
    

    
    
      
  • 7×7 depthwise conv (Swin-এর 7-window equivalent).
    • 
      
  • LayerNorm, GELU.
    • 
      
  • Inverted bottleneck.
    • 
      
  • Pure CNN, no attention.
    • 
      
  • Often beats Swin on similar setting.
    • 
    


    
১০ · Swin vs ConvNeXt vs ViT

    

      
        
ModelImageNetDetection (COCO)Mobile

      
      
        
ViT-B81.8%N/A directHard

        
Swin-B83.5%51.9Medium

        
ConvNeXt-B83.8%52.6Better

        
EfficientNet-B784.3%N/AYes

      
    


    

      Swin — २०२१-২০২২-এর dominant detection backbone. ২০২৪+ — ConvNeXt-V2, EVA, DINO-v2 backbone competing. Architecture race ongoing.
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      
        প্র ০১
        Window attention compute reduce, কিন্তু long-range dependency limit. Practical impact কী?
      
      

        
Window-attention trade-off — efficiency vs global context.

        
ViT global attention:

        
    
          
  • Layer 1 — every patch sees every patch.
    • 
          
  • Long-range dependency immediate.
    • 
          
  • Compute O(N²)।
    • 
        

        
Swin local attention:

        
    
          
  • Layer 1 — patch sees only window (49 patch).
    • 
          
  • Long-range dependency build through stages.
    • 
          
  • Stage 4 — receptive field eventually covers everything.
    • 
          
  • Compute linear.
    • 
        

        
Effective receptive field:

        
    
          
  • Stage 1: 7×7 window.
    • 
          
  • Stage 2: shifted + downsample → 14×14 effective.
    • 
          
  • Stage 4: 7×7 window-এ 32x downsample = 224×224 effective.
    • 
          
  • Full image cover by stage 4.
    • 
        

        
Practical impact:

        
    
          
  • Shape, size, fine-grained — Swin equally good.
    • 
          
  • Truly long-range (e.g., scene context cross-half image) — slightly worse than ViT.
    • 
          
  • Most CV task — local sufficient.
    • 
        

        
Mitigation:

        
    
          
  • Larger window (Swin v2 — 24×24).
    • 
          
  • Global attention layers occasionally.
    • 
          
  • Cross-window mechanisms.
    • 
        

        
Real comparison:

        
    
          
  • ImageNet — Swin within 0.5% of ViT.
    • 
          
  • Detection — Swin BETTER (multi-scale).
    • 
          
  • Segmentation — Swin BETTER.
    • 
          
  • Long-range reasoning task — ViT slight edge.
    • 
        

        
মূল উপলব্ধি: "Global attention always better" — myth. Most vision task local + hierarchy যথেষ্ট। Swin-এর efficiency win.

      

    


    

      
        প্র ০২
        Shifted window — implementation-এ tricky. Cyclic shift trick কী, padding/masking কীভাবে?
      
      

        
Shifted window-এর efficient implementation — engineering brilliance.

        
Naive shift:

        
    
          
  • Window positions shift M/2.
    • 
          
  • Edge windows partial — pad zero.
    • 
          
  • More windows (e.g., 9 instead of 4).
    • 
          
  • Compute increase.
    • 
        

        
Cyclic shift trick:

        
    
          
  • Image-কে cyclic shift (top→bottom, left→right wrap).
    • 
          
  • Windows now regular grid again.
    • 
          
  • Same compute as non-shifted.
    • 
          
  • Reverse shift after attention.
    • 
        

        
Masking:

        
    
          
  • Cyclic shift-এ — cross-image-boundary patches mixed.
    • 
          
  • Attention-এ mask — invalid pair prevent.
    • 
          
  • Mask pre-computed, reuse.
    • 
        

        
Implementation:

        
def shift_window(x, shift):
    # x: (B, H, W, C)
    return torch.roll(x, shifts=(-shift, -shift), dims=(1, 2))

def reverse_shift(x, shift):
    return torch.roll(x, shifts=(shift, shift), dims=(1, 2))

# In Swin block
if shift > 0:
    x = shift_window(x, shift)
    attn = window_attention(x, mask)
    x = reverse_shift(attn, shift)
else:
    x = window_attention(x)

        
Mask construction:

        
    
          
  • Per window — patches from different "regions" identify.
    • 
          
  • Within-region attention OK.
    • 
          
  • Cross-region attention masked.
    • 
          
  • Once compute, reuse for all forward.
    • 
        

        
Performance:

        
    
          
  • Cyclic + mask — same compute as non-shifted.
    • 
          
  • Memory minimal overhead.
    • 
          
  • Backward pass also efficient.
    • 
        

        
Lesson:

        
    
          
  • Algorithm correctness ≠ efficient implementation.
    • 
          
  • Hardware-aware design crucial.
    • 
          
  • Engineering art — research paper-এর underrated component.
    • 
        

        
মূল উপলব্ধি: Implementation cleverness — Swin-এর success-এর key. "Pretty algorithm" alone insufficient — efficient code matters.

      

    


    

      
        প্র ০৩
        Swin detection-এ champion ছিল ২০২১। ২০২৬-এ — DINO-DETR, Mask2Former এ Swin backbone vs full transformer? Trade-off?
      
      

        
Architecture wars-এর latest chapter.

        
Swin backbone:

        
    
          
  • Hierarchical features → FPN compatible.
    • 
          
  • Detection head choice flexible.
    • 
          
  • Training mature, code well-tested.
    • 
          
  • Mask R-CNN, DETR — both works.
    • 
        

        
ViT backbone (vanilla):

        
    
          
  • Single-scale — detection awkward.
    • 
          
  • Modify (e.g., ViTDet) make work.
    • 
          
  • Self-supervised pretrain (MAE) very strong.
    • 
        

        
DINO-DETR + Swin:

        
    
          
  • Swin backbone + DETR head.
    • 
          
  • COCO 63.3 mAP — top performer.
    • 
          
  • Proven combination.
    • 
        

        
EVA-02 + Mask2Former:

        
    
          
  • EVA pretrained ViT.
    • 
          
  • Mask2Former head.
    • 
          
  • Slightly better than Swin equivalent.
    • 
        

        
ConvNeXt + DINO:

        
    
          
  • Pure CNN backbone with modern training.
    • 
          
  • Mobile-friendlier.
    • 
          
  • Similar detection accuracy.
    • 
        

        
Compute trade-off:

        
    
          
  • Swin — moderate compute.
    • 
          
  • ViT-MAE — slightly more.
    • 
          
  • EVA — heavy.
    • 
          
  • ConvNeXt — efficient.
    • 
        

        
Production recommendation 2026:

        
    
          
  • SOTA accuracy: EVA-02 + Mask2Former.
    • 
          
  • Best speed/accuracy: Swin-B + DINO-DETR.
    • 
          
  • Mobile: ConvNeXt-Tiny + YOLO.
    • 
          
  • Realtime: RT-DETR (Swin/ResNet variants).
    • 
        

        
Bangladesh practical:

        
    
          
  • GPU-rich research — EVA-02.
    • 
          
  • Production server — Swin-B.
    • 
          
  • Edge — ConvNeXt-Tiny.
    • 
        

        
মূল উপলব্ধি: "Best architecture" task + compute + ecosystem-dependent. Swin still strong, ViT/CNN/hybrid-এর mix — modern reality.

      

    


    

      
        প্র ০৪
        Swin v2 (২০২২) 3B params — vision foundation model. Trillion-parameter vision আছে কি? Future direction?
      
      

        
Vision scaling — NLP-এর pace follow করছে।

        
Vision model size timeline:

        
    
          
  • ২০১২ AlexNet: 60M.
    • 
          
  • ২০১৭ ResNeXt-101: 84M.
    • 
          
  • ২০১৯ EfficientNet-B7: 66M.
    • 
          
  • ২০২০ ViT-Huge: 632M.
    • 
          
  • ২০২২ Swin-V2-3B: 3B.
    • 
          
  • ২০২২-২০২৩ ViT-22B (Google): 22B.
    • 
          
  • ২০২৩ EVA-02: 1B.
    • 
          
  • ২০২৪ DINOv2: ~1B.
    • 
        

        
Compared to LLM:

        
    
          
  • GPT-3: 175B.
    • 
          
  • GPT-4: speculatively 1T+.
    • 
          
  • Llama-3 405B.
    • 
          
  • Vision lagging by 100x.
    • 
        

        
Why slower scaling vision:

        
    
          
  • Image data scarcer than text (curated).
    • 
          
  • Cost — each image 100x text-bytes.
    • 
          
  • Pretraining objective less clear (MAE vs DINO vs supervised).
    • 
          
  • Less commercial pressure.
    • 
        

        
Multimodal era:

        
    
          
  • Vision + language combine — GPT-4V, Gemini.
    • 
          
  • Vision encoder typically smaller (1-10B).
    • 
          
  • Most "intelligence" in language part.
    • 
        

        
Where pure vision scale:

        
    
          
  • Medical imaging — DINOv2 + finetune.
    • 
          
  • Satellite/remote sensing — billion-scale.
    • 
          
  • Industrial — task-specific large model.
    • 
        

        
Future trends:

        
    
          
  • Multimodal native (image + video + text + audio).
    • 
          
  • Vision-language pretraining dominant.
    • 
          
  • Pure vision foundation model role narrow but persistent.
    • 
        

        
Bangladesh implication:

        
    
          
  • Train such model impossible locally.
    • 
          
  • Foundation model API (Anthropic, OpenAI) leverage.
    • 
          
  • Local fine-tune (LoRA) only realistic option.
    • 
          
  • Open foundation model (DINOv2, Llama) preferred.
    • 
        

        
মূল উপলব্ধি: Vision foundation model age dawning. Bangladesh — adapter, applier, evaluator. Research-funded scaling unattainable, application-layer rich.

      

    

  


  

    
 অনুশীলন

    
    
      
  1. 
        Compute compare: 224×224 image, patch 4 (Swin-T) → token count? Window 7×7 — windows কত?
        
    
          
          
    
            
    Token: $(224/4)^2 = 56^2 = 3136$। Window: $56/7 = 8$ per row, $8^2 = 64$ windows.
    
          
    
        
    
      
    2. 

      
  2. 
        Pretrained: torchvision Swin-T load and inference.
        
    
          
          
    
            
    from torchvision.models import swin_t
m = swin_t(weights='DEFAULT').eval()
out = m(torch.randn(1, 3, 224, 224))
    
          
    
        
    
      
    3. 

      
  3. 
        ভাবুন: Bangladesh medical 3D MRI segment — Swin-UNet vs U-Net কখন কোনটা?
        
    
          
          
    
            
    Small dataset (under 100 volumes) → U-Net. Large multi-hospital dataset (1000+) → Swin-UNet — long-range context capture better.
    
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন · ABCL TECH-এ আপনার পরবর্তী পদক্ষেপ

    
    

      কোড রানার কাজ না করলে? Google Colab use করুন।
    

  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ২৫ · Vision Transformer