পাঠ ২৮ · ৩৫-এর মধ্যে · মডিউল ৪
Home / AI Courses / Computer Vision / DETR

DETR — Detection Transformer

DETR — end-to-end transformer detection
৭ মিনিট পড়া উচ্চ · Advanced PyTorch কোডসহ

এই পাঠে যা শিখবেন

  • DETR architecture — encoder-decoder transformer
  • Object queries ও set prediction
  • Hungarian matching loss
  • DETR variants — Deformable, DINO, RT-DETR

১ · DETR-এর paradigm shift

Faster R-CNN, YOLO — heuristic-heavy: anchor design, NMS, IoU matching. Performance high but pipeline messy.

DETR — "Detection as set prediction". Transformer encoder-decoder. Fixed N output slot. Each slot → object বা "no object".

কেন্দ্রীয় ধারণা

DETR detection-কে set prediction problem হিসেবে treat করে। Hungarian matching ensure unique prediction per ground-truth. NMS, anchor — সব heuristic বাদ।

২ · DETR architecture

(ক) CNN backbone

  • ResNet-50/101 — feature extract.
  • Output: $H/32 \times W/32 \times 2048$।
  • Project to 256-D, flatten.

(খ) Transformer encoder

  • 6 layer self-attention.
  • Each spatial position attends all.
  • Global context capture.

(গ) Transformer decoder

  • 6 layer cross-attention.
  • Input: N (e.g., 100) "object queries" (learnable).
  • Each query → final embedding.

(ঘ) Prediction heads

  • Classification: per query → class (or "no object").
  • Bbox: per query → 4 coordinate.

৩ · Object queries

Each query = "slot" — ছবিতে একটি object look for. Learnable embedding.

  • Initialized random — train-এ specialize.
  • Empirical: each query learns "preferred location/size".
  • Query 1 — left corner small object.
  • Query 2 — center large object. Etc.

৪ · Hungarian matching

Training time — N predictions vs M ground-truth (N >> M). One-to-one matching:

  • Bipartite graph: predictions × ground-truths.
  • Edge cost: classification + bbox + IoU.
  • Hungarian algorithm — minimum cost matching.
  • Matched: predict GT.
  • Unmatched: predict "no object".

৫ · Loss function

$$\mathcal{L} = \mathcal{L}_{\text{cls}} + \lambda \mathcal{L}_{\text{bbox}} + \mu \mathcal{L}_{\text{IoU}}$$

  • $\mathcal{L}_{\text{cls}}$: matched class prediction.
  • $\mathcal{L}_{\text{bbox}}$: L1 box coordinate.
  • $\mathcal{L}_{\text{IoU}}$: generalized IoU loss.
  • Unmatched query: "no object" classification only.

৬ · Slow convergence problem

  • DETR original: 500 epoch on COCO.
  • Faster R-CNN: 12 epoch.
  • 40x slower train.

Why?

  • No anchor prior — model search from scratch.
  • Cross-attention slow to localize.
  • Hungarian matching unstable early.

৭ · Deformable DETR (২০২১)

Zhu et al. — sparse attention.

  • Each query attends only K (e.g., 4) reference point.
  • Learnable offset.
  • 10x faster convergence.
  • Multi-scale feature support.

৮ · DINO-DETR (২০২৩)

Zhang et al. — current SOTA detection.

  • Contrastive denoising training.
  • Mixed query selection.
  • Look-forward-twice update.
  • COCO 63.3 mAP — top accuracy.

৯ · RT-DETR (২০২৩, Baidu)

Realtime DETR — YOLOv8 competitor.

  • NMS-free → no NMS latency.
  • Efficient hybrid encoder.
  • 53% mAP, 100+ FPS.
  • Apache 2.0 license — startup-friendly.
Faster R-CNN = "rule-based assembly line". DETR = "smart workshop with N apprentices, each finds objects, no clashes by design". Less heuristic, more elegant.
DETR — set prediction transformer detector 📷 Image CNN ResNet TF Encoder 6 layer self-attn TF Decoder cross-attn × 6 N predictions class + bbox or ∅ N Object queries learnable, e.g., 100 Hungarian Matching one-to-one assignment Loss: cls + bbox + IoU no matching heuristic Direct set prediction — no anchor, no NMS, end-to-end
DETR — CNN feature → transformer encoder-decoder → N predictions. Object queries learn slot specialization.

১০ · PyTorch DETR

Python · DETR 
# pip install transformers
from transformers import DetrImageProcessor, DetrForObjectDetection
from PIL import Image
import torch

processor = DetrImageProcessor.from_pretrained("facebook/detr-resnet-50")
model = DetrForObjectDetection.from_pretrained("facebook/detr-resnet-50")

image = Image.open("photo.jpg")
inputs = processor(images=image, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)

# Convert to COCO API format
target_sizes = torch.tensor([image.size[::-1]])
results = processor.post_process_object_detection(
    outputs, target_sizes=target_sizes, threshold=0.5
)[0]

print("Boxes:", results["boxes"].shape)
print("Scores:", results["scores"])
print("Labels:", results["labels"])

    


    

      Hugging Face transformers — DETR easy use. ResNet-50 backbone, COCO pretrained. 100 query default.
    


    
১১ · DETR-derived models

    
    
      
  • Mask2Former: universal segmentation (semantic + instance + panoptic).
    • 
      
  • OWL-ViT: open-vocabulary detection.
    • 
      
  • Grounding DINO: text-prompted detection.
    • 
      
  • SAM: not exactly DETR but similar set prediction.
    • 
    


    

      DETR — research field-এর "Cambrian explosion". Faster R-CNN era ended ২০২০। ২০২৬-এ — DINO-DETR variants dominant detection benchmark.
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      
        প্র ০১
        DETR-এ "object queries" learnable. Visualize করলে কী দেখবেন? Each query কি specific?
      
      

        
DETR-এর interpretability surprise!

        
Empirical observation (paper-এ):

        
    
          
  • Each query develop "preferred region".
    • 
          
  • Some query — left side, large object.
    • 
          
  • Some query — center, small object.
    • 
          
  • Implicit specialization, despite no explicit supervision.
    • 
        

        
Visualization:

        
    
          
  • Run DETR on COCO val set.
    • 
          
  • For each query, plot center of all detections.
    • 
          
  • Distinct clusters per query.
    • 
          
  • Roughly matches "anchor" but learned.
    • 
        

        
Why specialize:

        
    
          
  • Hungarian matching — query 5 mostly assigned to bottom-left objects.
    • 
          
  • Gradient — query 5 specialize for that region.
    • 
          
  • Self-organizing.
    • 
        

        
Number of queries:

        
    
          
  • 100 default (DETR).
    • 
          
  • 300 (Deformable DETR).
    • 
          
  • 900 (DINO-DETR).
    • 
          
  • More queries — more capacity, but slower.
    • 
        

        
Image content adaptation:

        
    
          
  • Modern variants — content-aware query selection.
    • 
          
  • "Mixed query" — encoder feature derive query.
    • 
          
  • DINO-DETR-এর key innovation.
    • 
        

        
Practical implication:

        
    
          
  • Crowded scene — more queries needed.
    • 
          
  • Single object detection — small N enough.
    • 
          
  • Tunable for use case.
    • 
        

        
মূল উপলব্ধি: Implicit anchor learning. DETR's queries equivalent to learned anchor — but more flexible, content-adaptive.

      

    


    

      
        প্র ০২
        Hungarian matching algorithm — combinatorial optimization. Differentiable কীভাবে? Backprop possible?
      
      

        
Hungarian matching itself non-differentiable — clever workaround.

        
Hungarian algorithm:

        
    
          
  • Combinatorial assignment optimization.
    • 
          
  • O(n^3) time complexity.
    • 
          
  • Discrete output — assignment matrix.
    • 
          
  • Cannot differentiate directly.
    • 
        

        
Workaround in DETR:

        
    
          
  • Forward pass: compute cost matrix (cls + bbox + IoU).
    • 
          
  • Run Hungarian — get assignment.
    • 
          
  • Backward pass: use assignment as fixed labels.
    • 
          
  • Standard cross-entropy + L1 loss on assigned pair.
    • 
          
  • Gradient flows through cost computation, not algorithm.
    • 
        

        
Why this works:

        
    
          
  • Assignment binary — gradient through doesn't help.
    • 
          
  • Once assigned, treat as standard supervised learning.
    • 
          
  • Gradient-based optimization on (matched pair) loss.
    • 
        

        
Practical implementation:

        
from scipy.optimize import linear_sum_assignment

# In forward
cost_matrix = compute_cost(predictions, targets)
row_ind, col_ind = linear_sum_assignment(cost_matrix.detach().cpu())
# Use assignment for loss compute
loss = cross_entropy(predictions[row_ind], targets[col_ind].class) + ...

        
Alternative — Sinkhorn:

        
    
          
  • Soft Sinkhorn distance — differentiable approximation.
    • 
          
  • Some recent papers use.
    • 
          
  • DETR-এ Hungarian + detach standard.
    • 
        

        
Stability concern:

        
    
          
  • Early training — assignment unstable.
    • 
          
  • Same prediction may match different GT epoch-by-epoch.
    • 
          
  • Slow convergence partial reason.
    • 
        

        
Modern solutions:

        
    
          
  • Auxiliary losses — every decoder layer.
    • 
          
  • Denoising training — stable matching.
    • 
          
  • Many-to-one matching during training (Hybrid).
    • 
        

        
মূল উপলব্ধি: "Non-differentiable component + standard backprop on assigned pair" — common pattern in modern DL. RL-style assignment + supervised gradient.

      

    


    

      
        প্র ০৩
        RT-DETR YOLOv8-এর সমান speed-এ better mAP দাবি করে। ২০২৬-এ realtime detection-এ কোনটা winner?
      
      

        
২০২৩-পরবর্তী active battle.

        
Performance comparison:

        
    
          
  • YOLOv8-l: 52.9 mAP, 60 FPS.
    • 
          
  • RT-DETR-L: 53.0 mAP, 100 FPS.
    • 
          
  • RT-DETR slightly faster, similar accuracy.
    • 
        

        
RT-DETR advantages:

        
    
          
  • NMS-free — saves 2-5 ms post-process.
    • 
          
  • Cleaner pipeline.
    • 
          
  • Apache 2.0 license.
    • 
          
  • Better small object.
    • 
        

        
YOLOv8 advantages:

        
    
          
  • More mature ecosystem (Ultralytics).
    • 
          
  • Easier finetune.
    • 
          
  • Mobile deployment easier (ONNX, TFLite).
    • 
          
  • Larger community resources.
    • 
        

        
Mobile reality:

        
    
          
  • YOLOv8-nano — 280 FPS on T4.
    • 
          
  • RT-DETR mobile — 30-50 FPS.
    • 
          
  • YOLO mobile-friendlier still.
    • 
        

        
Bangladesh deployment:

        
    
          
  • Server CCTV — RT-DETR (cleaner, faster).
    • 
          
  • Mobile traffic — YOLOv8-nano.
    • 
          
  • Custom small dataset — YOLOv8 (mature).
    • 
        

        
Newer competitors (2024-2025):

        
    
          
  • D-FINE: claimed SOTA realtime.
    • 
          
  • YOLO11 (Ultralytics 2024): latest YOLO.
    • 
          
  • RTMDet: OpenMMLab competitor.
    • 
        

        
Verdict 2026:

        
    
          
  • No clear winner — close race.
    • 
          
  • Use case dictates choice.
    • 
          
  • Track latest benchmark, not last year's.
    • 
        

        
মূল উপলব্ধি: Realtime detection mature field — incremental progress. Architecture choice tactical, ecosystem strategic.

      

    


    

      
        প্র ০৪
        Grounding DINO — text prompt-এ object detect. কীভাবে? CLIP + DETR-এর marriage?
      
      

        
Grounding DINO — open-vocabulary detection-এর breakthrough.

        
Idea:

        
    
          
  • Closed-set detection: 80 COCO class.
    • 
          
  • Open-vocabulary: any object describable by text.
    • 
          
  • Text prompt → detection.
    • 
        

        
Architecture:

        
    
          
  • Vision backbone (Swin, ViT).
    • 
          
  • Text encoder (BERT/T5).
    • 
          
  • Cross-attention modules — text-image fusion.
    • 
          
  • DETR-style decoder.
    • 
          
  • Output: bbox aligned with text query.
    • 
        

        
Training data:

        
    
          
  • Existing detection dataset (COCO labels).
    • 
          
  • Phrase grounding (Flickr30K).
    • 
          
  • Image-text caption (LAION).
    • 
          
  • Multi-source — vocabulary diversity.
    • 
        

        
Inference example:

        
prompt = "a green apple"
image = load("kitchen.jpg")
boxes = grounding_dino(image, prompt)
# Returns boxes around green apples (only)

        
Beyond detection:

        
    
          
  • Combine with SAM → "segment anything described".
    • 
          
  • Grounded-SAM — language-prompted segmentation.
    • 
          
  • Annotation tool revolutionary.
    • 
        

        
Use cases:

        
    
          
  • Image search — natural language query.
    • 
          
  • Robot — "find the red mug".
    • 
          
  • Content moderation — "weapon", "alcohol".
    • 
          
  • Research — quick prototyping.
    • 
        

        
Limitations:

        
    
          
  • Compositional reasoning weak.
    • 
          
  • "Cat NOT on table" — negation struggle.
    • 
          
  • Spatial relation occasional miss.
    • 
          
  • Tail vocabulary inconsistent.
    • 
        

        
Bangladesh use:

        
    
          
  • Bangla prompt — translate to English first.
    • 
          
  • Custom finetune Bangladesh visual culture.
    • 
          
  • Educational tool — student question search.
    • 
        

        
মূল উপলব্ধি: Detection-এর foundation model era. Grounding DINO + SAM — annotation/search/robotics revolution.

      

    

  


  

    
 অনুশীলন

    
    
      
  1. 
        Hungarian cost: 3 prediction, 2 GT — cost matrix shape কী?
        
    
          
          
    
            
    (3, 2) — predictions × ground-truth. Pad with "no object" for square matrix if needed.
    
          
    
        
    
      
    2. 

      
  2. 
        Pretrained DETR: Hugging Face DETR inference example.
        
    
          
          
    
            
    Code section ১০-এ। RT-DETR alternative: PekingU/rtdetr_r50vd Hugging Face.
    
          
    
        
    
      
    3. 

      
  3. 
        ভাবুন: Bangladesh traffic (rickshaw, CNG, bus) — DETR vs YOLO কোনটা?
        
    
          
          
    
            
    Crowded — DETR/RT-DETR (no NMS issue). Realtime mobile — YOLOv8. Server processing — DINO-DETR for accuracy.
    
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন · ABCL TECH-এ আপনার পরবর্তী পদক্ষেপ

    
    

      কোড রানার কাজ না করলে? Google Colab use করুন।
    

  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ২৭ · CLIP