পাঠ ২৮ · ৩৫-এর মধ্যে · মডিউল ৪
Home / AI Courses / Computer Vision / DETR

DETR — Detection Transformer

DETR — end-to-end transformer detection
৭ মিনিট পড়া উচ্চ · Advanced PyTorch কোডসহ

এই পাঠে যা শিখবেন

  • DETR architecture — encoder-decoder transformer
  • Object queries ও set prediction
  • Hungarian matching loss
  • DETR variants — Deformable, DINO, RT-DETR

১ · DETR-এর paradigm shift

Faster R-CNN, YOLO — heuristic-heavy: anchor design, NMS, IoU matching। Performance high but pipeline messy।

DETR — "Detection as set prediction"। Transformer encoder-decoder। Fixed N output slot। Each slot → object বা "no object"।

কেন্দ্রীয় ধারণা

DETR detection-কে set prediction problem হিসেবে treat করে। Hungarian matching ensure unique prediction per ground-truth। NMS, anchor — সব heuristic বাদ।

২ · DETR architecture

(ক) CNN backbone

  • ResNet-50/101 — feature extract।
  • Output: $H/32 \times W/32 \times 2048$।
  • Project to 256-D, flatten।

(খ) Transformer encoder

  • 6 layer self-attention।
  • Each spatial position attends all।
  • Global context capture।

(গ) Transformer decoder

  • 6 layer cross-attention।
  • Input: N (e.g., 100) "object queries" (learnable)।
  • Each query → final embedding।

(ঘ) Prediction heads

  • Classification: per query → class (or "no object")।
  • Bbox: per query → 4 coordinate।

৩ · Object queries

Each query = "slot" — ছবিতে একটি object look for। Learnable embedding।

  • Initialized random — train-এ specialize।
  • Empirical: each query learns "preferred location/size"।
  • Query 1 — left corner small object।
  • Query 2 — center large object। Etc.

৪ · Hungarian matching

Training time — N predictions vs M ground-truth (N >> M)। One-to-one matching:

  • Bipartite graph: predictions × ground-truths।
  • Edge cost: classification + bbox + IoU।
  • Hungarian algorithm — minimum cost matching।
  • Matched: predict GT।
  • Unmatched: predict "no object"।

৫ · Loss function

$$\mathcal{L} = \mathcal{L}_{\text{cls}} + \lambda \mathcal{L}_{\text{bbox}} + \mu \mathcal{L}_{\text{IoU}}$$

  • $\mathcal{L}_{\text{cls}}$: matched class prediction।
  • $\mathcal{L}_{\text{bbox}}$: L1 box coordinate।
  • $\mathcal{L}_{\text{IoU}}$: generalized IoU loss।
  • Unmatched query: "no object" classification only।

৬ · Slow convergence problem

  • DETR original: 500 epoch on COCO।
  • Faster R-CNN: 12 epoch।
  • 40x slower train।

Why?

  • No anchor prior — model search from scratch।
  • Cross-attention slow to localize।
  • Hungarian matching unstable early।

৭ · Deformable DETR (২০২১)

Zhu et al. — sparse attention।

  • Each query attends only K (e.g., 4) reference point।
  • Learnable offset।
  • 10x faster convergence।
  • Multi-scale feature support।

৮ · DINO-DETR (২০২৩)

Zhang et al. — current SOTA detection।

  • Contrastive denoising training।
  • Mixed query selection।
  • Look-forward-twice update।
  • COCO 63.3 mAP — top accuracy।

৯ · RT-DETR (২০২৩, Baidu)

Realtime DETR — YOLOv8 competitor।

  • NMS-free → no NMS latency।
  • Efficient hybrid encoder।
  • 53% mAP, 100+ FPS।
  • Apache 2.0 license — startup-friendly।
Faster R-CNN = "rule-based assembly line"। DETR = "smart workshop with N apprentices, each finds objects, no clashes by design"। Less heuristic, more elegant।
DETR — set prediction transformer detector 📷 Image CNN ResNet TF Encoder 6 layer self-attn TF Decoder cross-attn × 6 N predictions class + bbox or ∅ N Object queries learnable, e.g., 100 Hungarian Matching one-to-one assignment Loss: cls + bbox + IoU no matching heuristic Direct set prediction — no anchor, no NMS, end-to-end
DETR — CNN feature → transformer encoder-decoder → N predictions। Object queries learn slot specialization।

১০ · PyTorch DETR

Python · DETR 
# pip install transformers
from transformers import DetrImageProcessor, DetrForObjectDetection
from PIL import Image
import torch

processor = DetrImageProcessor.from_pretrained("facebook/detr-resnet-50")
model = DetrForObjectDetection.from_pretrained("facebook/detr-resnet-50")

image = Image.open("photo.jpg")
inputs = processor(images=image, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)

# Convert to COCO API format
target_sizes = torch.tensor([image.size[::-1]])
results = processor.post_process_object_detection(
    outputs, target_sizes=target_sizes, threshold=0.5
)[0]

print("Boxes:", results["boxes"].shape)
print("Scores:", results["scores"])
print("Labels:", results["labels"])

    


    

      Hugging Face transformers — DETR easy use। ResNet-50 backbone, COCO pretrained। 100 query default।
    


    
১১ · DETR-derived models

    
    
      
  • Mask2Former: universal segmentation (semantic + instance + panoptic)।
    • 
      
  • OWL-ViT: open-vocabulary detection।
    • 
      
  • Grounding DINO: text-prompted detection।
    • 
      
  • SAM: not exactly DETR but similar set prediction।
    • 
    


    

      DETR — research field-এর "Cambrian explosion"। Faster R-CNN era ended ২০২০। ২০২৬-এ — DINO-DETR variants dominant detection benchmark।
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      
        প্র ০১
        DETR-এ "object queries" learnable। Visualize করলে কী দেখবেন? Each query কি specific?
      
      

        
DETR-এর interpretability surprise!

        
Empirical observation (paper-এ):

        
    
          
  • Each query develop "preferred region"।
    • 
          
  • Some query — left side, large object।
    • 
          
  • Some query — center, small object।
    • 
          
  • Implicit specialization, despite no explicit supervision।
    • 
        

        
Visualization:

        
    
          
  • Run DETR on COCO val set।
    • 
          
  • For each query, plot center of all detections।
    • 
          
  • Distinct clusters per query।
    • 
          
  • Roughly matches "anchor" but learned।
    • 
        

        
Why specialize:

        
    
          
  • Hungarian matching — query 5 mostly assigned to bottom-left objects।
    • 
          
  • Gradient — query 5 specialize for that region।
    • 
          
  • Self-organizing।
    • 
        

        
Number of queries:

        
    
          
  • 100 default (DETR)।
    • 
          
  • 300 (Deformable DETR)।
    • 
          
  • 900 (DINO-DETR)।
    • 
          
  • More queries — more capacity, but slower।
    • 
        

        
Image content adaptation:

        
    
          
  • Modern variants — content-aware query selection।
    • 
          
  • "Mixed query" — encoder feature derive query।
    • 
          
  • DINO-DETR-এর key innovation।
    • 
        

        
Practical implication:

        
    
          
  • Crowded scene — more queries needed।
    • 
          
  • Single object detection — small N enough।
    • 
          
  • Tunable for use case।
    • 
        

        
মূল উপলব্ধি: Implicit anchor learning। DETR's queries equivalent to learned anchor — but more flexible, content-adaptive।

      

    


    

      
        প্র ০২
        Hungarian matching algorithm — combinatorial optimization। Differentiable কীভাবে? Backprop possible?
      
      

        
Hungarian matching itself non-differentiable — clever workaround।

        
Hungarian algorithm:

        
    
          
  • Combinatorial assignment optimization।
    • 
          
  • O(n^3) time complexity।
    • 
          
  • Discrete output — assignment matrix।
    • 
          
  • Cannot differentiate directly।
    • 
        

        
Workaround in DETR:

        
    
          
  • Forward pass: compute cost matrix (cls + bbox + IoU)।
    • 
          
  • Run Hungarian — get assignment।
    • 
          
  • Backward pass: use assignment as fixed labels।
    • 
          
  • Standard cross-entropy + L1 loss on assigned pair।
    • 
          
  • Gradient flows through cost computation, not algorithm।
    • 
        

        
Why this works:

        
    
          
  • Assignment binary — gradient through doesn't help।
    • 
          
  • Once assigned, treat as standard supervised learning।
    • 
          
  • Gradient-based optimization on (matched pair) loss।
    • 
        

        
Practical implementation:

        
from scipy.optimize import linear_sum_assignment

# In forward
cost_matrix = compute_cost(predictions, targets)
row_ind, col_ind = linear_sum_assignment(cost_matrix.detach().cpu())
# Use assignment for loss compute
loss = cross_entropy(predictions[row_ind], targets[col_ind].class) + ...

        
Alternative — Sinkhorn:

        
    
          
  • Soft Sinkhorn distance — differentiable approximation।
    • 
          
  • Some recent papers use।
    • 
          
  • DETR-এ Hungarian + detach standard।
    • 
        

        
Stability concern:

        
    
          
  • Early training — assignment unstable।
    • 
          
  • Same prediction may match different GT epoch-by-epoch।
    • 
          
  • Slow convergence partial reason।
    • 
        

        
Modern solutions:

        
    
          
  • Auxiliary losses — every decoder layer।
    • 
          
  • Denoising training — stable matching।
    • 
          
  • Many-to-one matching during training (Hybrid)।
    • 
        

        
মূল উপলব্ধি: "Non-differentiable component + standard backprop on assigned pair" — common pattern in modern DL। RL-style assignment + supervised gradient।

      

    


    

      
        প্র ০৩
        RT-DETR YOLOv8-এর সমান speed-এ better mAP দাবি করে। ২০২৬-এ realtime detection-এ কোনটা winner?
      
      

        
২০২৩-পরবর্তী active battle।

        
Performance comparison:

        
    
          
  • YOLOv8-l: 52.9 mAP, 60 FPS।
    • 
          
  • RT-DETR-L: 53.0 mAP, 100 FPS।
    • 
          
  • RT-DETR slightly faster, similar accuracy।
    • 
        

        
RT-DETR advantages:

        
    
          
  • NMS-free — saves 2-5 ms post-process।
    • 
          
  • Cleaner pipeline।
    • 
          
  • Apache 2.0 license।
    • 
          
  • Better small object।
    • 
        

        
YOLOv8 advantages:

        
    
          
  • More mature ecosystem (Ultralytics)।
    • 
          
  • Easier finetune।
    • 
          
  • Mobile deployment easier (ONNX, TFLite)।
    • 
          
  • Larger community resources।
    • 
        

        
Mobile reality:

        
    
          
  • YOLOv8-nano — 280 FPS on T4।
    • 
          
  • RT-DETR mobile — 30-50 FPS।
    • 
          
  • YOLO mobile-friendlier still।
    • 
        

        
Bangladesh deployment:

        
    
          
  • Server CCTV — RT-DETR (cleaner, faster)।
    • 
          
  • Mobile traffic — YOLOv8-nano।
    • 
          
  • Custom small dataset — YOLOv8 (mature)।
    • 
        

        
Newer competitors (2024-2025):

        
    
          
  • D-FINE: claimed SOTA realtime।
    • 
          
  • YOLO11 (Ultralytics 2024): latest YOLO।
    • 
          
  • RTMDet: OpenMMLab competitor।
    • 
        

        
Verdict 2026:

        
    
          
  • No clear winner — close race।
    • 
          
  • Use case dictates choice।
    • 
          
  • Track latest benchmark, not last year's।
    • 
        

        
মূল উপলব্ধি: Realtime detection mature field — incremental progress। Architecture choice tactical, ecosystem strategic।

      

    


    

      
        প্র ০৪
        Grounding DINO — text prompt-এ object detect। কীভাবে? CLIP + DETR-এর marriage?
      
      

        
Grounding DINO — open-vocabulary detection-এর breakthrough।

        
Idea:

        
    
          
  • Closed-set detection: 80 COCO class।
    • 
          
  • Open-vocabulary: any object describable by text।
    • 
          
  • Text prompt → detection।
    • 
        

        
Architecture:

        
    
          
  • Vision backbone (Swin, ViT)।
    • 
          
  • Text encoder (BERT/T5)।
    • 
          
  • Cross-attention modules — text-image fusion।
    • 
          
  • DETR-style decoder।
    • 
          
  • Output: bbox aligned with text query।
    • 
        

        
Training data:

        
    
          
  • Existing detection dataset (COCO labels)।
    • 
          
  • Phrase grounding (Flickr30K)।
    • 
          
  • Image-text caption (LAION)।
    • 
          
  • Multi-source — vocabulary diversity।
    • 
        

        
Inference example:

        
prompt = "a green apple"
image = load("kitchen.jpg")
boxes = grounding_dino(image, prompt)
# Returns boxes around green apples (only)

        
Beyond detection:

        
    
          
  • Combine with SAM → "segment anything described"।
    • 
          
  • Grounded-SAM — language-prompted segmentation।
    • 
          
  • Annotation tool revolutionary।
    • 
        

        
Use cases:

        
    
          
  • Image search — natural language query।
    • 
          
  • Robot — "find the red mug"।
    • 
          
  • Content moderation — "weapon", "alcohol"।
    • 
          
  • Research — quick prototyping।
    • 
        

        
Limitations:

        
    
          
  • Compositional reasoning weak।
    • 
          
  • "Cat NOT on table" — negation struggle।
    • 
          
  • Spatial relation occasional miss।
    • 
          
  • Tail vocabulary inconsistent।
    • 
        

        
Bangladesh use:

        
    
          
  • Bangla prompt — translate to English first।
    • 
          
  • Custom finetune Bangladesh visual culture।
    • 
          
  • Educational tool — student question search।
    • 
        

        
মূল উপলব্ধি: Detection-এর foundation model era। Grounding DINO + SAM — annotation/search/robotics revolution।

      

    

  


  

    
 অনুশীলন

    
    
      
  1. 
        Hungarian cost: 3 prediction, 2 GT — cost matrix shape কী?
        
    
          
          
    
            
    (3, 2) — predictions × ground-truth। Pad with "no object" for square matrix if needed।
    
          
    
        
    
      
    2. 

      
  2. 
        Pretrained DETR: Hugging Face DETR inference example।
        
    
          
          
    
            
    Code section ১০-এ। RT-DETR alternative: PekingU/rtdetr_r50vd Hugging Face।
    
          
    
        
    
      
    3. 

      
  3. 
        ভাবুন: Bangladesh traffic (rickshaw, CNG, bus) — DETR vs YOLO কোনটা?
        
    
          
          
    
            
    Crowded — DETR/RT-DETR (no NMS issue)। Realtime mobile — YOLOv8। Server processing — DINO-DETR for accuracy।
    
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন · ABCL TECH-এ আপনার পরবর্তী পদক্ষেপ

    
    

      কোড রানার কাজ না করলে? Google Colab use করুন।
    

  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ২৭ · CLIP