পাঠ ২৪ · ৩৫-এর মধ্যে · মডিউল ৩
Home / AI Courses / Computer Vision / SAM

SAM — Segment Anything

Segment Anything Model — promptable foundation segmentation
৭ মিনিট পড়া উচ্চ · Advanced Python কোডসহ

এই পাঠে যা শিখবেন

  • SAM-এর foundation model approach
  • Prompt encoder, image encoder, mask decoder
  • SA-1B dataset construction
  • Practical use — annotation, interactive editing

১ · Foundation model in vision

LLM (GPT)-এর parallel — generic পাঠ থেকে specific task adapt। SAM CV-তে এই philosophy bring করেছে। "Single model for any segmentation"।

Authors: Alex Kirillov, Eric Mintun, Nikhila Ravi, et al. (Meta AI Research, এপ্রিল ২০২৩)।

কেন্দ্রীয় ধারণা

SAM = "promptable" segmentation। User prompt (click, box, text) দেয় → model object segment করে। "Train once, segment any object"।

২ · Three-component architecture

(ক) Image encoder — ViT-H

  • Vision Transformer Huge — 632M params।
  • MAE pretrained, then SA-1B fine-tuned।
  • Image → 64×64×256 embedding (dense feature map)।
  • Heavy compute — but per-image once।

(খ) Prompt encoder

  • Point: positional embedding + foreground/background label।
  • Box: top-left + bottom-right corner embed।
  • Text: CLIP encoder।
  • Mask: dense — convolutional encode।

(গ) Mask decoder

  • Lightweight transformer — 4M params।
  • Cross-attention: image embedding ↔ prompt embedding।
  • Output: 3 masks (handle ambiguity) + IoU score।
  • Fast — 50 ms per prompt after image encode।

৩ · Training data — SA-1B

  • 11 million image, 1.1 billion mask।
  • Average 100 mask per image।
  • Largest segmentation dataset ever।
  • Released openly।

Data collection three-stage:

  1. Manual: 50K image, 4M mask — human annotator।
  2. Semi-automatic: SAM v0 helps annotator — speed 5x।
  3. Fully automatic: SAM v1 generates mask, ambiguous cases human review।

৪ · Prompt types

  • Point click: 1-2 click → object mask। Most common।
  • Bounding box: rough box → tight mask।
  • Text: "cat" → cat mask (via CLIP)।
  • Mask: coarse mask → refined।
  • Mixed: click + box for finer control।

৫ · Ambiguity handling

Single click — ambiguous (whole person? face? eye?)। SAM output 3 masks at different granularity।

  • Mask 1: smallest (e.g., button on shirt)।
  • Mask 2: medium (shirt)।
  • Mask 3: largest (whole person)।
  • User pick — interactive workflow।

৬ · Zero-shot capability

SAM-এর key feature — training-এ unseen class segment। Why?

  • Massive diverse training data।
  • Prompt-based — class-agnostic।
  • "What's the boundary here?" — generic question।
  • Tested: medical image, satellite, microscopy — all work zero-shot।
Mask R-CNN = "Trained chef who knows 80 recipes (COCO classes)"। SAM = "Master who can cut any object given a point". Generic skill > specific knowledge।
SAM — promptable foundation segmentation 📷 Image 1024×1024 Image Encoder ViT-H (632M, slow) Image embedding 64×64×256 cached 🖱 Prompt point/box/text Prompt Encoder light, fast Mask Decoder 4M params, 50ms Mask 3 levels Image encoded once, prompt-mask pairs cheap Interactive — multiple click on same image fast
SAM — heavy image encoder once, lightweight prompt-decoder per click। Interactive use case ideal।

৭ · SAM use

Python · SAM 
# Install: pip install segment-anything
from segment_anything import sam_model_registry, SamPredictor
import cv2
import numpy as np

# Download checkpoint: sam_vit_h_4b8939.pth
sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
predictor = SamPredictor(sam)

# Load image
img = cv2.imread('photo.jpg')
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
predictor.set_image(img)              # encode once

# Click point on cat (e.g., 500, 300)
input_point = np.array([[500, 300]])
input_label = np.array([1])           # 1 = foreground

masks, scores, logits = predictor.predict(
    point_coords=input_point,
    point_labels=input_label,
    multimask_output=True,            # 3 masks
)

print("Masks shape:", masks.shape)    # (3, H, W)
print("Scores:", scores)
best = masks[scores.argmax()]
print(f"Best mask area: {best.sum()} pixels")

    


    

      Click → 3 mask + IoU score। User best-fit choose। Interactive annotation tool-এর foundation।
    


    
৮ · SAM use cases

    
    
      
  • Annotation acceleration: 10x faster mask annotation।
    • 
      
  • Image editing: Photoshop-like object selection।
    • 
      
  • Medical: radiologist click → tumor mask।
    • 
      
  • Robotics: grasp planning — object mask।
    • 
      
  • AR/VR: object isolation।
    • 
      
  • Video editing: SAM 2-এ track-and-mask।
    • 
      
  • Pre-processing for fine-tune: bootstrap labeled data।
    • 
    


    
৯ · SAM variants

    
    
      
  • SAM (২০২৩): original, ViT-H/L/B।
    • 
      
  • SAM 2 (২০২৪): video, memory mechanism।
    • 
      
  • MobileSAM, FastSAM: efficient inference।
    • 
      
  • SAM-Med2D, SAM-Med3D: medical-specific finetune।
    • 
      
  • HQ-SAM: high-quality refinement।
    • 
      
  • Grounded SAM: SAM + DINO — text-prompted detection + mask।
    • 
    


    
১০ · Limitations

    
    
      
  • Slow image encoder: ViT-H — 600 ms per image GPU।
    • 
      
  • No native classification: "what is this?" answer না।
    • 
      
  • Domain gap: medical, satellite — out-of-distribution case occasional miss।
    • 
      
  • Tiny object: single click for sub-10px object struggle।
    • 
      
  • No video native (SAM 1): SAM 2 fixes।
    • 
    


    

      SAM revolutionary annotation। কিন্তু classification তৈরি করে না — separate model দরকার class পেতে। Often: object detection (class) + SAM (mask) hybrid।
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      
        প্র ০১
        SAM-এর "promptable" paradigm — segmentation-এ revolutionary। NLP-এর GPT-এর সাথে কী similarity?
      
      

        
SAM Vision foundation model — Meta-র explicit goal NLP-এর GPT-style। Parallels deep।

        
Foundation model definition:

        
    
          
  • Massive data, generic pretrain।
    • 
          
  • Adaptable downstream।
    • 
          
  • Few-shot / zero-shot capability।
    • 
        

        
GPT vs SAM:

        
    
          
  • GPT: text prompt → text output।
    • 
          
  • SAM: visual prompt (click) → mask output।
    • 
          
  • Both: training task generic (LM next-token / promptable mask)।
    • 
        

        
"In-context learning" parallel:

        
    
          
  • GPT — few example in prompt → task adapt।
    • 
          
  • SAM — point click examples → object disambiguation।
    • 
        

        
Architecture echo:

        
    
          
  • Both transformer-based।
    • 
          
  • Both pretrain on massive data।
    • 
          
  • Both interactive use।
    • 
        

        
Difference:

        
    
          
  • GPT generative (text)।
    • 
          
  • SAM discriminative (mask)।
    • 
          
  • SAM no language reasoning native।
    • 
          
  • GPT no spatial vision native।
    • 
        

        
Research trajectory:

        
    
          
  • GPT-1 (২০১৮) → GPT-4 (2023) — 5 years।
    • 
          
  • SAM (২০২৩) → SAM 2 (২০২৪) — 1 year।
    • 
          
  • Vision foundation pace accelerating।
    • 
        

        
Future hybrid:

        
    
          
  • GPT-4V — vision input + language output।
    • 
          
  • Gemini — multimodal native।
    • 
          
  • Possibly: "SAM + GPT" multimodal foundation।
    • 
        

        
মূল উপলব্ধি: Foundation model paradigm — domain-agnostic। Vision lagging NLP, catching up। Bangladesh — multilingual + vision foundation needed।

      

    


    

      
        প্র ০২
        SA-1B dataset 1B mask — annotation কীভাবে সম্ভব? Cost ও quality trade-off?
      
      

        
SA-1B annotation — engineering marvel। 3-stage hybrid human-AI।

        
Stage 1 — Manual (4M masks):

        
    
          
  • Professional annotator।
    • 
          
  • Browser-based tool।
    • 
          
  • 14 sec average per mask।
    • 
          
  • 50K image।
    • 
          
  • Cost: ~$1M।
    • 
        

        
Stage 2 — Semi-automatic (5M masks):

        
    
          
  • SAM v0 trained on Stage 1 data।
    • 
          
  • Annotator: SAM proposes mask, human refines।
    • 
          
  • Speed 2x — 7 sec average।
    • 
          
  • Cost similar to Stage 1 per mask।
    • 
        

        
Stage 3 — Fully automatic (1B masks):

        
    
          
  • SAM v1 fine-tuned on Stage 1+2।
    • 
          
  • Generate mask candidates per grid point।
    • 
          
  • Filter low-quality (predicted IoU)।
    • 
          
  • Spot-check 0.1% mask।
    • 
          
  • Cost: only compute (negligible per mask)।
    • 
        

        
Image source:

        
    
          
  • Licensed photo (paid)।
    • 
          
  • 11M images।
    • 
          
  • Cost 11M × $0.05 = $550K।
    • 
        

        
Quality measures:

        
    
          
  • Stage 3 — model self-confidence।
    • 
          
  • Boundary smoothness check।
    • 
          
  • Stability across crops।
    • 
          
  • ~94% mask quality (vs full manual annotation)।
    • 
        

        
Total project cost:

        
    
          
  • Estimate $5-10M (image + annotation + compute)।
    • 
          
  • Compare: ImageNet $14M (২০০৯)।
    • 
          
  • Massive scale possible due to AI-assist।
    • 
        

        
Open license:

        
    
          
  • Apache 2.0 model।
    • 
          
  • Image — research only।
    • 
          
  • Substantial community gift।
    • 
        

        
মূল উপলব্ধি: Modern ML — data engineering primary cost, compute secondary। AI-assisted labeling — exponential scale enabled।

      

    


    

      
        প্র ০৩
        SAM medical imaging-এ zero-shot accuracy decent কিন্তু not SOTA। কেন? Domain-specific fine-tune (SAM-Med) কীভাবে gap close করে?
      
      

        
SAM domain transfer — interesting case study।

        
Zero-shot medical performance:

        
    
          
  • Natural image — 80%+ IoU click prompt।
    • 
          
  • Medical (CT, MRI) — 60-70%।
    • 
          
  • Microscopy cells — 40-60%।
    • 
          
  • Significant gap।
    • 
        

        
Why gap:

        
    
          
  • Domain shift: medical image grayscale, low contrast, anatomical structure ভিন্ন।
    • 
          
  • Boundary semantics ভিন্ন: "tumor edge" subjective, natural object edge clear।
    • 
          
  • Label noise tolerance: medical pixel-level perfect annotation rare।
    • 
          
  • Modality: 3D (CT/MRI), DICOM not natural RGB।
    • 
        

        
SAM-Med2D approach:

        
    
          
  • SAM-এর image encoder freeze (general feature)।
    • 
          
  • Prompt encoder + mask decoder finetune medical data।
    • 
          
  • 20+ medical dataset compiled।
    • 
          
  • +10-20% IoU improvement।
    • 
        

        
SAM-Med3D:

        
    
          
  • 3D extension — volumetric data।
    • 
          
  • Per-slice SAM unstable।
    • 
          
  • Cross-slice attention added।
    • 
          
  • CT, MRI volume-level segment।
    • 
        

        
MedSAM (Wang et al., 2024):

        
    
          
  • Specifically medical bounding box prompt।
    • 
          
  • 1.5M medical mask training।
    • 
          
  • Nature Communications paper।
    • 
          
  • FDA-pathway potential।
    • 
        

        
Adaptation strategy:

        
    
          
  • Light: few-shot prompt — same SAM, multiple click।
    • 
          
  • Medium: mask decoder finetune, encoder freeze।
    • 
          
  • Heavy: full finetune on domain data।
    • 
        

        
Bangladesh medical AI:

        
    
          
  • BIRDEM, ICDDR,B research projects।
    • 
          
  • SAM + Bengali radiologist clicks → annotation pipeline।
    • 
          
  • SAM-Med2D as baseline।
    • 
        

        
মূল উপলব্ধি: Foundation model — strong baseline, not always optimal। Domain-specific finetune still essential। Zero-shot promise + targeted improvement = practical pipeline।

      

    


    

      
        প্র ০৪
        Bangladesh annotation team-এ SAM use — workflow কী হবে? Cost saving estimate?
      
      

        
Real Bangladesh-এর use case।

        
Traditional annotation cost:

        
    
          
  • Bangladesh annotator: $0.50-2/hour।
    • 
          
  • Mask annotation: 30-60 sec per object।
    • 
          
  • 1000 image × 5 object × 45 sec = 62 hour = $30-120।
    • 
          
  • Quality check + revision: 2x time।
    • 
        

        
SAM-assisted workflow:

        
    
          
  1. Annotator click on object।
    2. 
          
  2. SAM 3 mask propose।
    3. 
          
  3. Annotator pick best (1 sec)।
    4. 
          
  4. Refine if needed (5-10 sec)।
    5. 
          
  5. Total: 5-15 sec per object।
    6. 
        

        
Speedup: 4-6x

        
Cost save:

        
    
          
  • 1000 image task: $30-120 → $5-25।
    • 
          
  • Per million mask: $30K-120K → $5K-25K।
    • 
          
  • Massive saving for big project।
    • 
        

        
Quality:

        
    
          
  • SAM mask often higher quality than rushed manual।
    • 
          
  • Boundary precision better।
    • 
          
  • Annotator focus on edge cases।
    • 
        

        
Required infrastructure:

        
    
          
  • GPU server (RTX 3060 sufficient): $300-1000।
    • 
          
  • Web tool (e.g., labelme, VIA, CVAT)।
    • 
          
  • SAM checkpoint (~2.5 GB)।
    • 
          
  • Internet acceptable (no continuous cloud need)।
    • 
        

        
Challenges:

        
    
          
  • Bangla-specific image — domain gap possible।
    • 
          
  • Annotator training on tool।
    • 
          
  • Edge case (overlap, transparent) — manual still।
    • 
        

        
Recommendation:

        
    
          
  • Hybrid pipeline: SAM proposal + human verify।
    • 
          
  • 2-stage: SAM auto + spot check।
    • 
          
  • Active learning: difficult cases priority।
    • 
        

        
Bangladesh CV company:

        
    
          
  • Annotation export competitive — local rate + AI-assist।
    • 
          
  • Global ML company (Scale AI competitor) — Bangladesh-based potential।
    • 
          
  • Skill: Python + image annotation + SAM operate।
    • 
        

        
মূল উপলব্ধি: SAM = Bangladesh annotation industry democratizer। Local tech-savvy team can offer global-quality service at competitive price।

      

    

  


  

    
 অনুশীলন

    
    
      
  1. 
        SAM install + run: SAM checkpoint download করে একটি image-এ click prompt।
        
    
          
          
    
            
    Download sam_vit_b_01ec64.pth (smaller, 375 MB)। Code পাঠের section ৭-এ।
    
          
    
        
    
      
    2. 

      
  2. 
        Multiple click: Foreground + background click combine — exclusion test।
        
    
          
          
    
            
    points = np.array([[500, 300], [600, 400]])  # foreground, then background
labels = np.array([1, 0])                     # 1=fg, 0=bg
masks, _, _ = predictor.predict(point_coords=points, point_labels=labels)
    
          
    
        
    
      
    3. 

      
  3. 
        ভাবুন: Bangla street sign annotation — SAM ব্যবহার করে annotation team-এর efficiency কত বাড়বে?
        
    
          
          
    
            
    Sign mostly square — SAM single click 95%+ accurate। 4-6x speedup। 1000 sign annotation: 8 hour → 2 hour।
    
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন · ABCL TECH-এ আপনার পরবর্তী পদক্ষেপ

    
    

      কোড রানার কাজ না করলে? Google Colab use করুন।
    

  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ২৩ · Mask R-CNN