পাঠ ২১ · ৩৫-এর মধ্যে · মডিউল ৩
Home / AI Courses / Computer Vision / Segmentation types

Semantic vs Instance segmentation

Segmentation types — semantic, instance, panoptic
৬ মিনিট পড়া মাঝারি · Intermediate PyTorch কোডসহ

এই পাঠে যা শিখবেন

  • তিন segmentation type-এর difference
  • "Stuff" vs "thing" categories
  • Architecture choice — encoder-decoder, Mask R-CNN, Mask2Former
  • Bangladesh use case — medical, satellite, autonomous

১ · Semantic segmentation

প্রতিটি pixel-এ class label. দু'টি পাশের কুকুর — দু'জনই "dog" পিক্সেল। Instance distinguish হয় না।

Use case: autonomous driving (road, sidewalk, building), satellite (water, vegetation), medical (organ).

Architecture: FCN (২০১৪), U-Net (২০১৫), DeepLab (২০১৬), SegFormer (২০২১), Segformer-Mask2Former (২০২২)।

Metric: mIoU (mean Intersection over Union per class).

২ · Instance segmentation

প্রতিটি object instance আলাদা mask. দু'টি কুকুর = দু'টি ভিন্ন mask.

Use case: robot grasping, cell counting, retail automation.

Architecture: Mask R-CNN (২০১৭), YOLACT, SOLO, Mask2Former.

Metric: AP (Average Precision over IoU thresholds).

৩ · Panoptic segmentation

Kirillov et al. (২০১৯) — semantic + instance unified.

  • "Stuff": uncountable — sky, road, grass. Semantic only.
  • "Thing": countable — car, person, dog. Instance distinguish.

Metric: PQ (Panoptic Quality).

কেন্দ্রীয় ধারণা

Detection bounding box-এ অ-tight. Segmentation pixel-precise. Per-pixel decision — বিরাট annotation effort, accurate output, applications more demanding.

৪ · Architecture overview

FCN (Fully Convolutional Network)

  • FC layer remove → all conv.
  • Upsample (transpose conv) — original resolution.
  • Output: per-pixel class probabilities.

Encoder-decoder

  • Encoder: downsample, abstract features (ResNet/VGG).
  • Decoder: upsample, refine to pixel level.
  • Skip connections — fine detail preserve.
  • Examples: U-Net, SegNet, LinkNet.

Atrous/Dilated convolution

  • Receptive field বাড়ায় without downsample.
  • DeepLab — atrous spatial pyramid pooling (ASPP).
  • Multi-scale context capture.

Transformer-based

  • Mask2Former, SegFormer — universal segmentation.
  • SAM — promptable segmentation.

৫ · Loss functions

  • Cross-entropy: per-pixel classification.
  • Dice loss: $1 - \frac{2|A \cap B|}{|A|+|B|}$ — class imbalance handle.
  • Focal loss: hard example focus.
  • Lovász loss: directly optimize IoU.
  • Combined: CE + Dice — robust.
Detection = "এই box-এ কী?"। Segmentation = "এই brushstroke-এ কী?" — pixel painter. Annotation 10-100x বেশি effort, output 10-100x বেশি detail.
Three segmentation paradigms Semantic all dogs same color per-pixel class only Instance dogs different color instance distinguish Panoptic sky (stuff) stuff + thing unified complexity ↑ · annotation cost ↑ · application richness ↑
তিন segmentation paradigm — প্রতিটির ভিন্ন output structure ও use case.

৬ · Pretrained model use

Python · PyTorch 
import torch
from torchvision.models.segmentation import deeplabv3_resnet50, DeepLabV3_ResNet50_Weights

# Pretrained DeepLab v3 — 21-class semantic segmentation (Pascal VOC)
w = DeepLabV3_ResNet50_Weights.COCO_WITH_VOC_LABELS_V1
model = deeplabv3_resnet50(weights=w)
model.eval()

x = torch.rand(1, 3, 480, 480)
with torch.no_grad():
    out = model(x)['out']
print("Output:", out.shape)         # (1, 21, 480, 480)

# Per-pixel class
mask = out.argmax(dim=1)            # (1, 480, 480)
print("Pixel class range:", mask.min().item(), mask.max().item())
print("Unique classes:", torch.unique(mask).tolist())

    


    

      Output shape: $(B, C, H, W)$ — per-pixel class score. Argmax → class map. Detection-এর চেয়ে dense output.
    


    
৭ · Datasets

    
    
      
  • PASCAL VOC: 21 class, 11K train. Classic benchmark.
    • 
      
  • Cityscapes: 19 class, 5K image, autonomous driving.
    • 
      
  • ADE20K: 150 class, 20K image, scene parsing.
    • 
      
  • COCO-Stuff: 172 class (80 thing + 92 stuff).
    • 
      
  • Mapillary Vistas: 124 class, street view.
    • 
    


    
৮ · Applications

    
    
      
  • Medical: tumor, organ segment. U-Net dominant.
    • 
      
  • Autonomous driving: drivable area, lane.
    • 
      
  • Satellite: land use, deforestation.
    • 
      
  • Agriculture: crop area, weed map.
    • 
      
  • Manufacturing: defect localize.
    • 
      
  • AR/VR: background remove, virtual try-on.
    • 
      
  • Photography: portrait mode bokeh.
    • 
    


    

      Segmentation annotation extremely expensive — pixel-level mask 30-90 minute per image. Active learning, weakly supervised, foundation model (SAM) — annotation effort reduce দিচ্ছে।
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      
        প্র ০১
        "Stuff" (sky, road) ও "thing" (car, person) distinction segmentation-এর philosophy কেন matter?
      
      

        
এই terminology Heitz & Koller (২০০৮) থেকে। CV community-র formal নাম।

        
"Things" — countable, bounded:

        
    
          
  • Car, person, dog — discrete instances.
    • 
          
  • Each instance separate identity.
    • 
          
  • Bounding box natural.
    • 
          
  • Detection task fits.
    • 
        

        
"Stuff" — uncountable, amorphous:

        
    
          
  • Sky, road, water, grass — boundary fluid.
    • 
          
  • "How many sky?" meaningless.
    • 
          
  • Per-pixel class natural.
    • 
          
  • Semantic segmentation fits.
    • 
        

        
Why categorize:

        
    
          
  • Annotation strategy ভিন্ন।
    • 
          
  • Loss function ভিন্ন (instance-aware vs not).
    • 
          
  • Architecture optimize ভিন্ন।
    • 
        

        
Edge cases:

        
    
          
  • Crowd of people — "thing" but instance unclear.
    • 
          
  • Forest of trees — single tree "thing", forest "stuff"?
    • 
          
  • Skin — "thing" of person বা "stuff" surface?
    • 
        

        
Panoptic unification:

        
    
          
  • Both treated under unified framework.
    • 
          
  • Single model output both.
    • 
          
  • Mask2Former — universal architecture.
    • 
        

        
Application implications:

        
    
          
  • Autonomous driving: car (thing) count, road (stuff) area.
    • 
          
  • Medical: tumor (thing) instance, healthy tissue (stuff).
    • 
          
  • Satellite: building (thing), forest (stuff).
    • 
        

        
মূল উপলব্ধি: Computer vision-এর ontology — natural language-এ borrow. Linguistic distinction (count vs mass noun) → CV task design influence.

      

    


    

      
        প্র ০২
        Segmentation annotation extremely expensive. Weak supervision, semi-supervised — কী options?
      
      

        
Pixel-level annotation cost — segmentation-এর primary blocker. Solutions evolving.

        
Annotation cost reality:

        
    
          
  • Cityscapes: average 1.5 hr per image.
    • 
          
  • Medical (radiologist): $50-200/image.
    • 
          
  • 5000 image dataset → multi-million dollar.
    • 
        

        
Weakly supervised:

        
    
          
  • Image-level label only: "this image contains cat". CAM (Class Activation Map) — region inference.
    • 
          
  • Bounding box only: easier to annotate. Predict mask within box.
    • 
          
  • Scribble: few brush strokes — extend to mask.
    • 
          
  • Point clicks: 1-2 points per object.
    • 
        

        
Semi-supervised:

        
    
          
  • Small labeled + large unlabeled.
    • 
          
  • Self-training: pseudo-label confident prediction.
    • 
          
  • Consistency regularization: same image augmented prediction match.
    • 
        

        
Foundation models:

        
    
          
  • SAM (Segment Anything): click → mask. Annotation tool revolutionary.
    • 
          
  • SAM2 (২০২৪): video extension.
    • 
          
  • Annotation 10x speedup.
    • 
        

        
Synthetic data:

        
    
          
  • 3D simulation (Unreal, Carla) — perfect annotation.
    • 
          
  • Domain gap challenge.
    • 
          
  • Sim-to-real transfer technique mature হচ্ছে।
    • 
        

        
Bangladesh-specific:

        
    
          
  • Local annotation team build.
    • 
          
  • Crowdsourcing platform (Toloka, Scale AI).
    • 
          
  • Active learning — most informative sample priority.
    • 
          
  • SAM-assisted annotation workflow.
    • 
        

        
মূল উপলব্ধি: Annotation = bottleneck. Smart annotation strategy data scientist priority. Foundation model era — annotation cost dramatically dropping.

      

    


    

      
        প্র ০৩
        mIoU ও PQ — segmentation-এর primary metric. কীভাবে compute, কী trade-off ধরে?
      
      

        
Metric design — segmentation evaluation-এর foundation.

        
IoU (per class):

        
$\text{IoU}_c = \frac{|P_c \cap G_c|}{|P_c \cup G_c|}$

        
    
          
  • $P_c$ = predicted pixel of class c.
    • 
          
  • $G_c$ = ground-truth.
    • 
          
  • 1.0 perfect, 0 no overlap.
    • 
        

        
mIoU:

        
    
          
  • $\text{mIoU} = \frac{1}{C} \sum_c \text{IoU}_c$।
    • 
          
  • Class-balanced — rare class equal weight.
    • 
          
  • Per-class breakdown important.
    • 
        

        
Pixel accuracy (alternative):

        
    
          
  • Total correct / total pixel.
    • 
          
  • Class imbalance hide — sky 90% pixel — high accuracy easy.
    • 
          
  • Naive baseline.
    • 
        

        
PQ (Panoptic Quality):

        
$\text{PQ} = \frac{\sum_{(p,g) \in TP} \text{IoU}(p, g)}{|TP| + 0.5|FP| + 0.5|FN|}$

        
    
          
  • TP = matched prediction (IoU > 0.5).
    • 
          
  • FP = unmatched prediction.
    • 
          
  • FN = unmatched ground-truth.
    • 
          
  • Two factor: matching quality (numerator) + recognition quality (denominator).
    • 
        

        
Decomposition:

        
    
          
  • $\text{PQ} = \text{SQ} \times \text{RQ}$।
    • 
          
  • $\text{SQ}$ — average IoU of matched.
    • 
          
  • $\text{RQ}$ — F1-like score.
    • 
        

        
Per-task metric:

        
    
          
  • Semantic: mIoU.
    • 
          
  • Instance: AP (COCO-style).
    • 
          
  • Panoptic: PQ.
    • 
        

        
Trade-offs:

        
    
          
  • mIoU — class equal weight (good for imbalance).
    • 
          
  • Pixel acc — fast, intuitive but imbalance-blind.
    • 
          
  • PQ — comprehensive but complex.
    • 
        

        
Practical use:

        
    
          
  • Production tracking — pixel accuracy + per-class breakdown.
    • 
          
  • Research — mIoU/PQ standard.
    • 
          
  • Domain-specific: Dice (medical), F1 (binary).
    • 
        

        
মূল উপলব্ধি: Single metric oversimplify. Multiple lens-এ deeply understand. Per-class analysis critical.

      

    


    

      
        প্র ০৪
        Segmentation-এ class imbalance severe (background dominant). Loss function কীভাবে handle? Dice vs Focal vs CE.
      
      

        
Class imbalance — segmentation-এর central problem. 99% background, 1% tumor — naive CE useless.

        
Cross-entropy (vanilla):

        
    
          
  • Per-pixel CE.
    • 
          
  • Background dominant gradient.
    • 
          
  • Foreground often missed.
    • 
          
  • Used যখন balance acceptable.
    • 
        

        
Weighted CE:

        
    
          
  • Per-class weight inverse to frequency.
    • 
          
  • $w_c = 1 / N_c$ or median frequency balance.
    • 
          
  • Hyperparameter sensitive.
    • 
        

        
Focal loss (Lin et al., 2017):

        
$\text{FL} = -\alpha (1 - p)^\gamma \log p$

        
    
          
  • $(1-p)^\gamma$ — confident prediction-এ loss reduce.
    • 
          
  • Hard example focus.
    • 
          
  • $\gamma = 2$ typical.
    • 
          
  • Originally object detection.
    • 
        

        
Dice loss:

        
$\mathcal{L}_{\text{Dice}} = 1 - \frac{2 \sum p_i g_i}{\sum p_i + \sum g_i}$

        
    
          
  • Class imbalance robust — ratio-based.
    • 
          
  • Direct IoU-like optimization.
    • 
          
  • Medical imaging gold standard.
    • 
          
  • Single value per class.
    • 
        

        
Tversky loss:

        
    
          
  • Dice generalization — FP/FN weight asymmetric.
    • 
          
  • $\alpha, \beta$ — false positive/negative weight.
    • 
          
  • Clinical recall priority — $\beta > \alpha$।
    • 
        

        
Combined approach (best practice):

        
    
          
  • $\mathcal{L} = \mathcal{L}_{\text{CE}} + \mathcal{L}_{\text{Dice}}$।
    • 
          
  • CE — pixel-level confidence.
    • 
          
  • Dice — class balance robust.
    • 
          
  • Most modern segmentation networks.
    • 
        

        
Lovász loss:

        
    
          
  • Direct IoU surrogate.
    • 
          
  • Submodular optimization.
    • 
          
  • Cityscapes top performer.
    • 
        

        
Dataset-specific:

        
    
          
  • Cityscapes (balanced) — CE OK.
    • 
          
  • Medical tumor (imbalanced) — Dice + Tversky.
    • 
          
  • Boundary refinement — boundary loss.
    • 
        

        
Practical recipe:

        
    
          
  • Start: CE + Dice weighted.
    • 
          
  • Imbalance severe → Focal added.
    • 
          
  • Boundary issue → boundary loss component.
    • 
        

        
মূল উপলব্ধি: Loss function design — segmentation-এ critical. Single CE often fails. Domain-aware combination essential.

      

    

  


  

    
 অনুশীলন

    
    
      
  1. 
        IoU calc: Predicted mask 90 pixel, ground-truth 100 pixel, intersection 80 — IoU?
        
    
          
          
    
            
    Union = $90 + 100 - 80 = 110$. IoU = $80/110 ≈ 0.73$।
    
          
    
        
    
      
    2. 

      
  2. 
        Pretrained: SegFormer থেকে ADE20K pretrain inference example.
        
    
          
          
    
            
    from transformers import SegformerForSemanticSegmentation
model = SegformerForSemanticSegmentation.from_pretrained(
    "nvidia/segformer-b0-finetuned-ade-512-512"
).eval()
# inputs through SegformerImageProcessor
    
          
    
        
    
      
    3. 

      
  3. 
        ভাবুন: Bangladesh agriculture — drone থেকে field crop area measure (rice, jute, vegetable). Semantic, instance, না panoptic?
        
    
          
          
    
            
    Semantic — crop type "stuff", area-of-interest (not individual plant). Instance over-engineered. Panoptic if both crop type ও individual tree (e.g., orchard).
    
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন · ABCL TECH-এ আপনার পরবর্তী পদক্ষেপ

    

    

      কোড রানার কাজ না করলে? ব্রাউজারে কাজ না করলে
      Google Colab
      ব্যবহার করুন।
    

  







  
  
     এই পাঠে ভুল পেয়েছেন? জানান
  





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ২০ · Anchor & NMS