পাঠ ২৪ · ৩৫-এর মধ্যে · মডিউল ৩

Home / AI Courses / Computer Vision / SAM

SAM — Segment Anything

Segment Anything Model — promptable foundation segmentation

৭ মিনিট পড়া উচ্চ · Advanced Python কোডসহ

এই পাঠে যা শিখবেন

SAM-এর foundation model approach
Prompt encoder, image encoder, mask decoder
SA-1B dataset construction
Practical use — annotation, interactive editing

১ · Foundation model in vision

LLM (GPT)-এর parallel — generic পাঠ থেকে specific task adapt. SAM CV-তে এই philosophy bring করেছে। "Single model for any segmentation".

Authors: Alex Kirillov, Eric Mintun, Nikhila Ravi, et al. (Meta AI Research, এপ্রিল ২০২৩)।

কেন্দ্রীয় ধারণা

SAM = "promptable" segmentation. User prompt (click, box, text) দেয় → model object segment করে। "Train once, segment any object".

২ · Three-component architecture

(ক) Image encoder — ViT-H

Vision Transformer Huge — 632M params.
MAE pretrained, then SA-1B fine-tuned.
Image → 64×64×256 embedding (dense feature map).
Heavy compute — but per-image once.

(খ) Prompt encoder

Point: positional embedding + foreground/background label.
Box: top-left + bottom-right corner embed.
Text: CLIP encoder.
Mask: dense — convolutional encode.

(গ) Mask decoder

Lightweight transformer — 4M params.
Cross-attention: image embedding ↔ prompt embedding.
Output: 3 masks (handle ambiguity) + IoU score.
Fast — 50 ms per prompt after image encode.

৩ · Training data — SA-1B

11 million image, 1.1 billion mask.
Average 100 mask per image.
Largest segmentation dataset ever.
Released openly.

Data collection three-stage:

Manual: 50K image, 4M mask — human annotator.
Semi-automatic: SAM v0 helps annotator — speed 5x.
Fully automatic: SAM v1 generates mask, ambiguous cases human review.

৪ · Prompt types

Point click: 1-2 click → object mask. Most common.
Bounding box: rough box → tight mask.
Text: "cat" → cat mask (via CLIP).
Mask: coarse mask → refined.
Mixed: click + box for finer control.

৫ · Ambiguity handling

Single click — ambiguous (whole person? face? eye?)। SAM output 3 masks at different granularity.

Mask 1: smallest (e.g., button on shirt).
Mask 2: medium (shirt).
Mask 3: largest (whole person).
User pick — interactive workflow.

৬ · Zero-shot capability

SAM-এর key feature — training-এ unseen class segment. Why?

Massive diverse training data.
Prompt-based — class-agnostic.
"What's the boundary here?" — generic question.
Tested: medical image, satellite, microscopy — all work zero-shot.

Mask R-CNN = "Trained chef who knows 80 recipes (COCO classes)". SAM = "Master who can cut any object given a point". Generic skill > specific knowledge.

SAM — heavy image encoder once, lightweight prompt-decoder per click. Interactive use case ideal.

৭ · SAM use

Python · SAM

# Install: pip install segment-anything
from segment_anything import sam_model_registry, SamPredictor
import cv2
import numpy as np

# Download checkpoint: sam_vit_h_4b8939.pth
sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
predictor = SamPredictor(sam)

# Load image
img = cv2.imread('photo.jpg')
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
predictor.set_image(img)              # encode once

# Click point on cat (e.g., 500, 300)
input_point = np.array([[500, 300]])
input_label = np.array([1])           # 1 = foreground

masks, scores, logits = predictor.predict(
    point_coords=input_point,
    point_labels=input_label,
    multimask_output=True,            # 3 masks
)

print("Masks shape:", masks.shape)    # (3, H, W)
print("Scores:", scores)
best = masks[scores.argmax()]
print(f"Best mask area: {best.sum()} pixels")

Click → 3 mask + IoU score. User best-fit choose. Interactive annotation tool-এর foundation.

৮ · SAM use cases

Annotation acceleration: 10x faster mask annotation.
Image editing: Photoshop-like object selection.
Medical: radiologist click → tumor mask.
Robotics: grasp planning — object mask.
AR/VR: object isolation.
Video editing: SAM 2-এ track-and-mask.
Pre-processing for fine-tune: bootstrap labeled data.

৯ · SAM variants

SAM (২০২৩): original, ViT-H/L/B.
SAM 2 (২০২৪): video, memory mechanism.
MobileSAM, FastSAM: efficient inference.
SAM-Med2D, SAM-Med3D: medical-specific finetune.
HQ-SAM: high-quality refinement.
Grounded SAM: SAM + DINO — text-prompted detection + mask.

১০ · Limitations

Slow image encoder: ViT-H — 600 ms per image GPU.
No native classification: "what is this?" answer না।
Domain gap: medical, satellite — out-of-distribution case occasional miss.
Tiny object: single click for sub-10px object struggle.
No video native (SAM 1): SAM 2 fixes.

SAM revolutionary annotation. কিন্তু classification তৈরি করে না — separate model দরকার class পেতে। Often: object detection (class) + SAM (mask) hybrid.

ভাবনার প্রশ্ন

প্র ০১ SAM-এর "promptable" paradigm — segmentation-এ revolutionary. NLP-এর GPT-এর সাথে কী similarity?

SAM Vision foundation model — Meta-র explicit goal NLP-এর GPT-style. Parallels deep.

Foundation model definition:

Massive data, generic pretrain.
Adaptable downstream.
Few-shot / zero-shot capability.

GPT vs SAM:

GPT: text prompt → text output.
SAM: visual prompt (click) → mask output.
Both: training task generic (LM next-token / promptable mask).

"In-context learning" parallel:

GPT — few example in prompt → task adapt.
SAM — point click examples → object disambiguation.

Architecture echo:

Both transformer-based.
Both pretrain on massive data.
Both interactive use.

Difference:

GPT generative (text).
SAM discriminative (mask).
SAM no language reasoning native.
GPT no spatial vision native.

Research trajectory:

GPT-1 (২০১৮) → GPT-4 (2023) — 5 years.
SAM (২০২৩) → SAM 2 (২০২৪) — 1 year.
Vision foundation pace accelerating.

Future hybrid:

GPT-4V — vision input + language output.
Gemini — multimodal native.
Possibly: "SAM + GPT" multimodal foundation.

মূল উপলব্ধি: Foundation model paradigm — domain-agnostic. Vision lagging NLP, catching up. Bangladesh — multilingual + vision foundation needed.

প্র ০২ SA-1B dataset 1B mask — annotation কীভাবে সম্ভব? Cost ও quality trade-off?

SA-1B annotation — engineering marvel. 3-stage hybrid human-AI.

Stage 1 — Manual (4M masks):

Professional annotator.
Browser-based tool.
14 sec average per mask.
50K image.
Cost: ~$1M.

Stage 2 — Semi-automatic (5M masks):

SAM v0 trained on Stage 1 data.
Annotator: SAM proposes mask, human refines.
Speed 2x — 7 sec average.
Cost similar to Stage 1 per mask.

Stage 3 — Fully automatic (1B masks):

SAM v1 fine-tuned on Stage 1+2.
Generate mask candidates per grid point.
Filter low-quality (predicted IoU).
Spot-check 0.1% mask.
Cost: only compute (negligible per mask).

Image source:

Licensed photo (paid).
11M images.
Cost 11M × $0.05 = $550K.

Quality measures:

Stage 3 — model self-confidence.
Boundary smoothness check.
Stability across crops.
~94% mask quality (vs full manual annotation).

Total project cost:

Estimate $5-10M (image + annotation + compute).
Compare: ImageNet $14M (২০০৯)।
Massive scale possible due to AI-assist.

Open license:

Apache 2.0 model.
Image — research only.
Substantial community gift.

মূল উপলব্ধি: Modern ML — data engineering primary cost, compute secondary. AI-assisted labeling — exponential scale enabled.

প্র ০৩ SAM medical imaging-এ zero-shot accuracy decent কিন্তু not SOTA. কেন? Domain-specific fine-tune (SAM-Med) কীভাবে gap close করে?

SAM domain transfer — interesting case study.

Zero-shot medical performance:

Natural image — 80%+ IoU click prompt.
Medical (CT, MRI) — 60-70%.
Microscopy cells — 40-60%.
Significant gap.

Why gap:

Domain shift: medical image grayscale, low contrast, anatomical structure ভিন্ন।
Boundary semantics ভিন্ন: "tumor edge" subjective, natural object edge clear.
Label noise tolerance: medical pixel-level perfect annotation rare.
Modality: 3D (CT/MRI), DICOM not natural RGB.

SAM-Med2D approach:

SAM-এর image encoder freeze (general feature).
Prompt encoder + mask decoder finetune medical data.
20+ medical dataset compiled.
+10-20% IoU improvement.

SAM-Med3D:

3D extension — volumetric data.
Per-slice SAM unstable.
Cross-slice attention added.
CT, MRI volume-level segment.

MedSAM (Wang et al., 2024):

Specifically medical bounding box prompt.
1.5M medical mask training.
Nature Communications paper.
FDA-pathway potential.

Adaptation strategy:

Light: few-shot prompt — same SAM, multiple click.
Medium: mask decoder finetune, encoder freeze.
Heavy: full finetune on domain data.

Bangladesh medical AI:

BIRDEM, ICDDR,B research projects.
SAM + Bengali radiologist clicks → annotation pipeline.
SAM-Med2D as baseline.

মূল উপলব্ধি: Foundation model — strong baseline, not always optimal. Domain-specific finetune still essential. Zero-shot promise + targeted improvement = practical pipeline.

প্র ০৪ Bangladesh annotation team-এ SAM use — workflow কী হবে? Cost saving estimate?

Real Bangladesh-এর use case.

Traditional annotation cost:

Bangladesh annotator: $0.50-2/hour.
Mask annotation: 30-60 sec per object.
1000 image × 5 object × 45 sec = 62 hour = $30-120.
Quality check + revision: 2x time.

SAM-assisted workflow:

Annotator click on object.
SAM 3 mask propose.
Annotator pick best (1 sec).
Refine if needed (5-10 sec).
Total: 5-15 sec per object.

Speedup: 4-6x

Cost save:

1000 image task: $30-120 → $5-25.
Per million mask: $30K-120K → $5K-25K.
Massive saving for big project.

Quality:

SAM mask often higher quality than rushed manual.
Boundary precision better.
Annotator focus on edge cases.

Required infrastructure:

GPU server (RTX 3060 sufficient): $300-1000.
Web tool (e.g., labelme, VIA, CVAT).
SAM checkpoint (~2.5 GB).
Internet acceptable (no continuous cloud need).

Challenges:

Bangla-specific image — domain gap possible.
Annotator training on tool.
Edge case (overlap, transparent) — manual still.

Recommendation:

Hybrid pipeline: SAM proposal + human verify.
2-stage: SAM auto + spot check.
Active learning: difficult cases priority.

Bangladesh CV company:

Annotation export competitive — local rate + AI-assist.
Global ML company (Scale AI competitor) — Bangladesh-based potential.
Skill: Python + image annotation + SAM operate.

মূল উপলব্ধি: SAM = Bangladesh annotation industry democratizer. Local tech-savvy team can offer global-quality service at competitive price.

অনুশীলন

SAM install + run: SAM checkpoint download করে একটি image-এ click prompt.

Download sam_vit_b_01ec64.pth (smaller, 375 MB). Code পাঠের section ৭-এ।

Multiple click: Foreground + background click combine — exclusion test.

points = np.array([[500, 300], [600, 400]])  # foreground, then background
labels = np.array([1, 0])                     # 1=fg, 0=bg
masks, _, _ = predictor.predict(point_coords=points, point_labels=labels)

ভাবুন: Bangla street sign annotation — SAM ব্যবহার করে annotation team-এর efficiency কত বাড়বে?

Sign mostly square — SAM single click 95%+ accurate. 4-6x speedup. 1000 sign annotation: 8 hour → 2 hour.