পাঠ ২১ · ৩৫-এর মধ্যে · মডিউল ৩
Home / AI Courses / Computer Vision / Segmentation types

Semantic vs Instance segmentation

Segmentation types — semantic, instance, panoptic
৬ মিনিট পড়া মাঝারি · Intermediate PyTorch কোডসহ

এই পাঠে যা শিখবেন

  • তিন segmentation type-এর difference
  • "Stuff" vs "thing" categories
  • Architecture choice — encoder-decoder, Mask R-CNN, Mask2Former
  • Bangladesh use case — medical, satellite, autonomous

১ · Semantic segmentation

প্রতিটি pixel-এ class label। দু'টি পাশের কুকুর — দু'জনই "dog" পিক্সেল। Instance distinguish হয় না।

Use case: autonomous driving (road, sidewalk, building), satellite (water, vegetation), medical (organ).

Architecture: FCN (২০১৪), U-Net (২০১৫), DeepLab (২০১৬), SegFormer (২০২১), Segformer-Mask2Former (২০২২)।

Metric: mIoU (mean Intersection over Union per class)।

২ · Instance segmentation

প্রতিটি object instance আলাদা mask। দু'টি কুকুর = দু'টি ভিন্ন mask।

Use case: robot grasping, cell counting, retail automation।

Architecture: Mask R-CNN (২০১৭), YOLACT, SOLO, Mask2Former।

Metric: AP (Average Precision over IoU thresholds)।

৩ · Panoptic segmentation

Kirillov et al. (২০১৯) — semantic + instance unified।

  • "Stuff": uncountable — sky, road, grass। Semantic only।
  • "Thing": countable — car, person, dog। Instance distinguish।

Metric: PQ (Panoptic Quality)।

কেন্দ্রীয় ধারণা

Detection bounding box-এ অ-tight। Segmentation pixel-precise। Per-pixel decision — বিরাট annotation effort, accurate output, applications more demanding।

৪ · Architecture overview

FCN (Fully Convolutional Network)

  • FC layer remove → all conv।
  • Upsample (transpose conv) — original resolution।
  • Output: per-pixel class probabilities।

Encoder-decoder

  • Encoder: downsample, abstract features (ResNet/VGG)।
  • Decoder: upsample, refine to pixel level।
  • Skip connections — fine detail preserve।
  • Examples: U-Net, SegNet, LinkNet।

Atrous/Dilated convolution

  • Receptive field বাড়ায় without downsample।
  • DeepLab — atrous spatial pyramid pooling (ASPP)।
  • Multi-scale context capture।

Transformer-based

  • Mask2Former, SegFormer — universal segmentation।
  • SAM — promptable segmentation।

৫ · Loss functions

  • Cross-entropy: per-pixel classification।
  • Dice loss: $1 - \frac{2|A \cap B|}{|A|+|B|}$ — class imbalance handle।
  • Focal loss: hard example focus।
  • Lovász loss: directly optimize IoU।
  • Combined: CE + Dice — robust।
Detection = "এই box-এ কী?"। Segmentation = "এই brushstroke-এ কী?" — pixel painter। Annotation 10-100x বেশি effort, output 10-100x বেশি detail।
Three segmentation paradigms Semantic all dogs same color per-pixel class only Instance dogs different color instance distinguish Panoptic sky (stuff) stuff + thing unified complexity ↑ · annotation cost ↑ · application richness ↑
তিন segmentation paradigm — প্রতিটির ভিন্ন output structure ও use case।

৬ · Pretrained model use

Python · PyTorch 
import torch
from torchvision.models.segmentation import deeplabv3_resnet50, DeepLabV3_ResNet50_Weights

# Pretrained DeepLab v3 — 21-class semantic segmentation (Pascal VOC)
w = DeepLabV3_ResNet50_Weights.COCO_WITH_VOC_LABELS_V1
model = deeplabv3_resnet50(weights=w)
model.eval()

x = torch.rand(1, 3, 480, 480)
with torch.no_grad():
    out = model(x)['out']
print("Output:", out.shape)         # (1, 21, 480, 480)

# Per-pixel class
mask = out.argmax(dim=1)            # (1, 480, 480)
print("Pixel class range:", mask.min().item(), mask.max().item())
print("Unique classes:", torch.unique(mask).tolist())

    


    

      Output shape: $(B, C, H, W)$ — per-pixel class score। Argmax → class map। Detection-এর চেয়ে dense output।
    


    
৭ · Datasets

    
    
      
  • PASCAL VOC: 21 class, 11K train। Classic benchmark।
    • 
      
  • Cityscapes: 19 class, 5K image, autonomous driving।
    • 
      
  • ADE20K: 150 class, 20K image, scene parsing।
    • 
      
  • COCO-Stuff: 172 class (80 thing + 92 stuff)।
    • 
      
  • Mapillary Vistas: 124 class, street view।
    • 
    


    
৮ · Applications

    
    
      
  • Medical: tumor, organ segment। U-Net dominant।
    • 
      
  • Autonomous driving: drivable area, lane।
    • 
      
  • Satellite: land use, deforestation।
    • 
      
  • Agriculture: crop area, weed map।
    • 
      
  • Manufacturing: defect localize।
    • 
      
  • AR/VR: background remove, virtual try-on।
    • 
      
  • Photography: portrait mode bokeh।
    • 
    


    

      Segmentation annotation extremely expensive — pixel-level mask 30-90 minute per image। Active learning, weakly supervised, foundation model (SAM) — annotation effort reduce দিচ্ছে।
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      
        প্র ০১
        "Stuff" (sky, road) ও "thing" (car, person) distinction segmentation-এর philosophy কেন matter?
      
      

        
এই terminology Heitz & Koller (২০০৮) থেকে। CV community-র formal নাম।

        
"Things" — countable, bounded:

        
    
          
  • Car, person, dog — discrete instances।
    • 
          
  • Each instance separate identity।
    • 
          
  • Bounding box natural।
    • 
          
  • Detection task fits।
    • 
        

        
"Stuff" — uncountable, amorphous:

        
    
          
  • Sky, road, water, grass — boundary fluid।
    • 
          
  • "How many sky?" meaningless।
    • 
          
  • Per-pixel class natural।
    • 
          
  • Semantic segmentation fits।
    • 
        

        
Why categorize:

        
    
          
  • Annotation strategy ভিন্ন।
    • 
          
  • Loss function ভিন্ন (instance-aware vs not)।
    • 
          
  • Architecture optimize ভিন্ন।
    • 
        

        
Edge cases:

        
    
          
  • Crowd of people — "thing" but instance unclear।
    • 
          
  • Forest of trees — single tree "thing", forest "stuff"?
    • 
          
  • Skin — "thing" of person বা "stuff" surface?
    • 
        

        
Panoptic unification:

        
    
          
  • Both treated under unified framework।
    • 
          
  • Single model output both।
    • 
          
  • Mask2Former — universal architecture।
    • 
        

        
Application implications:

        
    
          
  • Autonomous driving: car (thing) count, road (stuff) area।
    • 
          
  • Medical: tumor (thing) instance, healthy tissue (stuff)।
    • 
          
  • Satellite: building (thing), forest (stuff)।
    • 
        

        
মূল উপলব্ধি: Computer vision-এর ontology — natural language-এ borrow। Linguistic distinction (count vs mass noun) → CV task design influence।

      

    


    

      
        প্র ০২
        Segmentation annotation extremely expensive। Weak supervision, semi-supervised — কী options?
      
      

        
Pixel-level annotation cost — segmentation-এর primary blocker। Solutions evolving।

        
Annotation cost reality:

        
    
          
  • Cityscapes: average 1.5 hr per image।
    • 
          
  • Medical (radiologist): $50-200/image।
    • 
          
  • 5000 image dataset → multi-million dollar।
    • 
        

        
Weakly supervised:

        
    
          
  • Image-level label only: "this image contains cat"। CAM (Class Activation Map) — region inference।
    • 
          
  • Bounding box only: easier to annotate। Predict mask within box।
    • 
          
  • Scribble: few brush strokes — extend to mask।
    • 
          
  • Point clicks: 1-2 points per object।
    • 
        

        
Semi-supervised:

        
    
          
  • Small labeled + large unlabeled।
    • 
          
  • Self-training: pseudo-label confident prediction।
    • 
          
  • Consistency regularization: same image augmented prediction match।
    • 
        

        
Foundation models:

        
    
          
  • SAM (Segment Anything): click → mask। Annotation tool revolutionary।
    • 
          
  • SAM2 (২০২৪): video extension।
    • 
          
  • Annotation 10x speedup।
    • 
        

        
Synthetic data:

        
    
          
  • 3D simulation (Unreal, Carla) — perfect annotation।
    • 
          
  • Domain gap challenge।
    • 
          
  • Sim-to-real transfer technique mature হচ্ছে।
    • 
        

        
Bangladesh-specific:

        
    
          
  • Local annotation team build।
    • 
          
  • Crowdsourcing platform (Toloka, Scale AI)।
    • 
          
  • Active learning — most informative sample priority।
    • 
          
  • SAM-assisted annotation workflow।
    • 
        

        
মূল উপলব্ধি: Annotation = bottleneck। Smart annotation strategy data scientist priority। Foundation model era — annotation cost dramatically dropping।

      

    


    

      
        প্র ০৩
        mIoU ও PQ — segmentation-এর primary metric। কীভাবে compute, কী trade-off ধরে?
      
      

        
Metric design — segmentation evaluation-এর foundation।

        
IoU (per class):

        
$\text{IoU}_c = \frac{|P_c \cap G_c|}{|P_c \cup G_c|}$

        
    
          
  • $P_c$ = predicted pixel of class c।
    • 
          
  • $G_c$ = ground-truth।
    • 
          
  • 1.0 perfect, 0 no overlap।
    • 
        

        
mIoU:

        
    
          
  • $\text{mIoU} = \frac{1}{C} \sum_c \text{IoU}_c$।
    • 
          
  • Class-balanced — rare class equal weight।
    • 
          
  • Per-class breakdown important।
    • 
        

        
Pixel accuracy (alternative):

        
    
          
  • Total correct / total pixel।
    • 
          
  • Class imbalance hide — sky 90% pixel — high accuracy easy।
    • 
          
  • Naive baseline।
    • 
        

        
PQ (Panoptic Quality):

        
$\text{PQ} = \frac{\sum_{(p,g) \in TP} \text{IoU}(p, g)}{|TP| + 0.5|FP| + 0.5|FN|}$

        
    
          
  • TP = matched prediction (IoU > 0.5)।
    • 
          
  • FP = unmatched prediction।
    • 
          
  • FN = unmatched ground-truth।
    • 
          
  • Two factor: matching quality (numerator) + recognition quality (denominator)।
    • 
        

        
Decomposition:

        
    
          
  • $\text{PQ} = \text{SQ} \times \text{RQ}$।
    • 
          
  • $\text{SQ}$ — average IoU of matched।
    • 
          
  • $\text{RQ}$ — F1-like score।
    • 
        

        
Per-task metric:

        
    
          
  • Semantic: mIoU।
    • 
          
  • Instance: AP (COCO-style)।
    • 
          
  • Panoptic: PQ।
    • 
        

        
Trade-offs:

        
    
          
  • mIoU — class equal weight (good for imbalance)।
    • 
          
  • Pixel acc — fast, intuitive but imbalance-blind।
    • 
          
  • PQ — comprehensive but complex।
    • 
        

        
Practical use:

        
    
          
  • Production tracking — pixel accuracy + per-class breakdown।
    • 
          
  • Research — mIoU/PQ standard।
    • 
          
  • Domain-specific: Dice (medical), F1 (binary)।
    • 
        

        
মূল উপলব্ধি: Single metric oversimplify। Multiple lens-এ deeply understand। Per-class analysis critical।

      

    


    

      
        প্র ০৪
        Segmentation-এ class imbalance severe (background dominant)। Loss function কীভাবে handle? Dice vs Focal vs CE।
      
      

        
Class imbalance — segmentation-এর central problem। 99% background, 1% tumor — naive CE useless।

        
Cross-entropy (vanilla):

        
    
          
  • Per-pixel CE।
    • 
          
  • Background dominant gradient।
    • 
          
  • Foreground often missed।
    • 
          
  • Used যখন balance acceptable।
    • 
        

        
Weighted CE:

        
    
          
  • Per-class weight inverse to frequency।
    • 
          
  • $w_c = 1 / N_c$ or median frequency balance।
    • 
          
  • Hyperparameter sensitive।
    • 
        

        
Focal loss (Lin et al., 2017):

        
$\text{FL} = -\alpha (1 - p)^\gamma \log p$

        
    
          
  • $(1-p)^\gamma$ — confident prediction-এ loss reduce।
    • 
          
  • Hard example focus।
    • 
          
  • $\gamma = 2$ typical।
    • 
          
  • Originally object detection।
    • 
        

        
Dice loss:

        
$\mathcal{L}_{\text{Dice}} = 1 - \frac{2 \sum p_i g_i}{\sum p_i + \sum g_i}$

        
    
          
  • Class imbalance robust — ratio-based।
    • 
          
  • Direct IoU-like optimization।
    • 
          
  • Medical imaging gold standard।
    • 
          
  • Single value per class।
    • 
        

        
Tversky loss:

        
    
          
  • Dice generalization — FP/FN weight asymmetric।
    • 
          
  • $\alpha, \beta$ — false positive/negative weight।
    • 
          
  • Clinical recall priority — $\beta > \alpha$।
    • 
        

        
Combined approach (best practice):

        
    
          
  • $\mathcal{L} = \mathcal{L}_{\text{CE}} + \mathcal{L}_{\text{Dice}}$।
    • 
          
  • CE — pixel-level confidence।
    • 
          
  • Dice — class balance robust।
    • 
          
  • Most modern segmentation networks।
    • 
        

        
Lovász loss:

        
    
          
  • Direct IoU surrogate।
    • 
          
  • Submodular optimization।
    • 
          
  • Cityscapes top performer।
    • 
        

        
Dataset-specific:

        
    
          
  • Cityscapes (balanced) — CE OK।
    • 
          
  • Medical tumor (imbalanced) — Dice + Tversky।
    • 
          
  • Boundary refinement — boundary loss।
    • 
        

        
Practical recipe:

        
    
          
  • Start: CE + Dice weighted।
    • 
          
  • Imbalance severe → Focal added।
    • 
          
  • Boundary issue → boundary loss component।
    • 
        

        
মূল উপলব্ধি: Loss function design — segmentation-এ critical। Single CE often fails। Domain-aware combination essential।

      

    

  


  

    
 অনুশীলন

    
    
      
  1. 
        IoU calc: Predicted mask 90 pixel, ground-truth 100 pixel, intersection 80 — IoU?
        
    
          
          
    
            
    Union = $90 + 100 - 80 = 110$. IoU = $80/110 ≈ 0.73$।
    
          
    
        
    
      
    2. 

      
  2. 
        Pretrained: SegFormer থেকে ADE20K pretrain inference example।
        
    
          
          
    
            
    from transformers import SegformerForSemanticSegmentation
model = SegformerForSemanticSegmentation.from_pretrained(
    "nvidia/segformer-b0-finetuned-ade-512-512"
).eval()
# inputs through SegformerImageProcessor
    
          
    
        
    
      
    3. 

      
  3. 
        ভাবুন: Bangladesh agriculture — drone থেকে field crop area measure (rice, jute, vegetable)। Semantic, instance, না panoptic?
        
    
          
          
    
            
    Semantic — crop type "stuff", area-of-interest (not individual plant)। Instance over-engineered। Panoptic if both crop type ও individual tree (e.g., orchard)।
    
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন · ABCL TECH-এ আপনার পরবর্তী পদক্ষেপ

    

    

      কোড রানার কাজ না করলে? ব্রাউজারে কাজ না করলে
      Google Colab
      ব্যবহার করুন।
    

  







  
  
     এই পাঠে ভুল পেয়েছেন? জানান
  





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ২০ · Anchor & NMS