পাঠ ২৭ · ৩৫-এর মধ্যে · মডিউল ৪
Home / AI Courses / Computer Vision / CLIP

CLIP — vision-language

CLIP — Contrastive Language–Image Pretraining
৮ মিনিট পড়া উচ্চ · Advanced Python কোডসহ

এই পাঠে যা শিখবেন

  • CLIP architecture — dual encoder
  • Contrastive learning (InfoNCE loss)
  • Zero-shot classification
  • Multimodal applications

১ · Vision-language alignment

Traditional CV — fixed class set (ImageNet 1000)। Real world — open vocabulary। "What's in this image?" answer infinite।

CLIP-এর insight: natural language-ই caption supervision। Web-এ billions of image-text pair available।

কেন্দ্রীয় ধারণা

CLIP = "image embedding ও text embedding-কে align করা"। Same embedding space-এ image ও তার description close, mismatched far। Zero-shot transfer enabled।

২ · CLIP architecture

(ক) Image encoder

  • ResNet বা ViT।
  • Image → feature vector (512-D for ViT-B)।
  • Linear projection → shared embedding space।

(খ) Text encoder

  • Transformer (GPT-style)।
  • Text → token embedding → final token vector (512-D)।
  • Linear projection → shared embedding space।

(গ) Shared 512-D space

  • Image embedding ও text embedding একই dimension।
  • Cosine similarity compare করা যায়।

৩ · Contrastive learning

Training-এ batch (e.g., 32K) image-text pair। Aim: matched pair close, mismatched far।

InfoNCE loss:

$$\mathcal{L} = -\frac{1}{N} \sum_i \log \frac{\exp(\text{sim}(I_i, T_i) / \tau)}{\sum_j \exp(\text{sim}(I_i, T_j) / \tau)}$$

  • $\text{sim}$ — cosine similarity।
  • $\tau$ — temperature (learnable)।
  • Positive: $(I_i, T_i)$ matched।
  • Negative: $(I_i, T_{j \ne i})$ — N-1 mismatched।
  • Symmetric loss — text-to-image direction also।

৪ · Training data

  • WIT (Web Image Text) — 400M image-text pair।
  • Crawled internet — caption, alt-text, etc।
  • Diverse — open vocabulary capture।
  • Noisy — কিন্তু scale-এ overcome।

৫ · Zero-shot classification

No fine-tune। Just rephrase classification as caption matching:

  1. Class names: ["cat", "dog", "car"]।
  2. Build caption: "A photo of a cat", "A photo of a dog", ...।
  3. Encode all caption — text embeddings।
  4. Encode image — image embedding।
  5. Cosine similarity image vs each caption।
  6. Max similarity → predicted class।

Result: ImageNet zero-shot — 76.2% top-1 (CLIP ViT-L)। Without ImageNet training!

Pre-CLIP: "I trained on cats and dogs only — anything else, I don't know"। CLIP: "I learnt language-image relationship — describe anything in words, I can find it visually"।
CLIP — dual encoder contrastive learning 📷 Image cat photo Image Encoder ViT-L (300M) image emb 512-D 📝 Text "a photo of a cat" Text Encoder Transformer (63M) text emb 512-D Cosine sim → matched/not Training: matched (cat ↔ "cat photo") high, mismatched low InfoNCE loss — softmax over batch negatives 400M image-text pair, web-scraped Zero-shot transfer to any classification by text prompt
CLIP — image ও text encoder একই embedding space-এ। Contrastive loss দিয়ে align।

৬ · CLIP use

Python · CLIP 
# pip install git+https://github.com/openai/CLIP.git
import clip
import torch
from PIL import Image

device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-L/14", device=device)

# Image preprocess
img = preprocess(Image.open("photo.jpg")).unsqueeze(0).to(device)

# Text candidates
texts = clip.tokenize([
    "a photo of a cat",
    "a photo of a dog",
    "a photo of a car"
]).to(device)

# Compute features
with torch.no_grad():
    image_features = model.encode_image(img)
    text_features = model.encode_text(texts)

# Cosine similarity
image_features = image_features / image_features.norm(dim=-1, keepdim=True)
text_features = text_features / text_features.norm(dim=-1, keepdim=True)
similarity = (image_features @ text_features.T).softmax(dim=-1)

print("Probabilities:", similarity[0].tolist())
# e.g., cat=0.92, dog=0.06, car=0.02

    


    

      Zero-shot — no training, no labels। Just text descriptions। CLIP understanding language-image association।
    


    
৭ · CLIP applications

    
    
      
  • Zero-shot classification: any class via text prompt।
    • 
      
  • Image search: "find sunset over mountain" → matching image।
    • 
      
  • Stable Diffusion: text condition encoder।
    • 
      
  • DALL-E: generation conditioning।
    • 
      
  • Open-vocabulary detection: Grounding DINO, OWL-ViT।
    • 
      
  • Image captioning: CLIP feature → LM generate caption।
    • 
      
  • Content moderation: "adult content" probability।
    • 
    


    
৮ · Multilingual CLIP

    
    
      
  • M-CLIP: 50+ languages। Bangla supported।
    • 
      
  • OpenCLIP: open implementation, multilingual variant।
    • 
      
  • SigLIP (২০২৩): sigmoid loss replace softmax — multilingual better।
    • 
      
  • BLIP, BLIP-2: generative + understanding।
    • 
    


    
৯ · CLIP limitations

    
    
      
  • Counting weakness: "3 cats" — CLIP struggles।
    • 
      
  • Spatial reasoning: "cat to left of dog" — limited।
    • 
      
  • Bias: web-scraped data — gender/race bias।
    • 
      
  • Fine-grained: bird species — coarse only।
    • 
      
  • Compositional: "red cube on blue sphere" — wrong sometimes।
    • 
    


    
১০ · CLIP variants

    
    
      
  • ALIGN (Google): 1B noisy image-text pair।
    • 
      
  • BASIC, FILIP: token-level alignment।
    • 
      
  • SigLIP: sigmoid loss।
    • 
      
  • EVA-CLIP: EVA backbone।
    • 
      
  • BLIP-2: Q-Former — better captioning।
    • 
    


    

      CLIP-এর success-এর key — scale + diverse data। Bangladesh-এ training cost prohibitive — pretrained CLIP/SigLIP use। Bangla specific finetune সম্ভব ছোট cost-এ।
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      
        প্র ০১
        "A photo of a cat" — কেন এই specific prompt? "Cat" alone কাজ করে কিন্তু কম accurate। Prompt engineering CV-তে?
      
      

        
CLIP-এর underrated detail — prompt template choice। "Prompt engineering" CV-তেও matters।

        
Why "A photo of":

        
    
          
  • Training data — captions look like "a photo of X"।
    • 
          
  • Model learnt this pattern।
    • 
          
  • Distribution match।
    • 
        

        
Effect on accuracy:

        
    
          
  • "cat" — 70%।
    • 
          
  • "a photo of a cat" — 76%।
    • 
          
  • "a photo of a {label}, a type of {category}" — 78%।
    • 
          
  • +8% from prompt alone!
    • 
        

        
Prompt ensembling:

        
    
          
  • Multiple prompt: "a photo of {}", "a picture of {}", "an image of {}"।
    • 
          
  • Average embeddings।
    • 
          
  • Robust।
    • 
        

        
Domain-specific:

        
    
          
  • Medical: "a chest X-ray showing {}"।
    • 
          
  • Satellite: "a satellite photo of {}"।
    • 
          
  • Drawing: "a sketch of {}"।
    • 
        

        
Bangla challenge:

        
    
          
  • OpenCLIP Bangla support — "একটি {} ছবি"।
    • 
          
  • Mixed: "a photo of {} (a {})" — English category + Bangla।
    • 
          
  • Test multiple, see what works।
    • 
        

        
Auto prompt:

        
    
          
  • CoOp (২০২২): learnable prompt token।
    • 
          
  • CoCoOp: conditional on image।
    • 
          
  • Adapt to dataset-specific।
    • 
        

        
মূল উপলব্ধি: Prompt = vital interface। Multimodal model-এ language-CV bridge careful design।

      

    


    

      
        প্র ০২
        CLIP zero-shot ImageNet 76% — supervised-এর কাছাকাছি। কীভাবে natural language supervision-এ এত strong?
      
      

        
CLIP-এর core insight — language label-এর চেয়ে rich।

        
ImageNet label:

        
    
          
  • 1 word/short phrase per image।
    • 
          
  • Discrete category।
    • 
          
  • 1000 class fixed।
    • 
        

        
Web caption:

        
    
          
  • Variable length sentence।
    • 
          
  • Continuous natural language।
    • 
          
  • Open vocabulary।
    • 
          
  • Compositional (color + object + action)।
    • 
        

        
Information density:

        
    
          
  • Caption "fluffy orange tabby cat sitting on red carpet" — much more signal than "cat"।
    • 
          
  • Multi-label implicit।
    • 
          
  • Spatial, attribute, context all encode।
    • 
        

        
Scale advantage:

        
    
          
  • ImageNet 1.4M label-image pair।
    • 
          
  • CLIP 400M image-text pair।
    • 
          
  • 300x more data + richer label।
    • 
        

        
Generalization:

        
    
          
  • Web-scraped — natural distribution।
    • 
          
  • Long-tail visual concept।
    • 
          
  • Adapts to new task without retraining।
    • 
        

        
Empirical:

        
    
          
  • CLIP linear probe ImageNet — 86%।
    • 
          
  • Above ResNet supervised on ImageNet itself।
    • 
          
  • Out-of-distribution robustness — far better।
    • 
        

        
Cost:

        
    
          
  • CLIP train — multi-million dollar GPU।
    • 
          
  • One time investment, broad applicability।
    • 
          
  • OpenAI-Anthropic-Google cost-affordable।
    • 
        

        
মূল উপলব্ধি: Label-এর nature matters। "Natural language as supervision" — paradigm shift।

      

    


    

      
        প্র ০৩
        Bangla CLIP — Bangladesh data-এর জন্য কী option? OpenCLIP মতো resources-এ Bangla আছে?
      
      

        
Multilingual CLIP — Bangla support gradually improving।

        
Available options:

        
    
          
  • M-CLIP (Multilingual CLIP): 50+ language, Bangla included। OpenAI CLIP-এর extension।
    • 
          
  • OpenCLIP: Apache 2.0 license, open weight। Multilingual variants।
    • 
          
  • SigLIP: Google, multilingual better generally।
    • 
          
  • NLLB-CLIP: Meta, 200+ language including Bangla।
    • 
        

        
Bangla-specific challenges:

        
    
          
  • Web data — Bangla less crawled than English।
    • 
          
  • Caption quality variable।
    • 
          
  • Code-mixed (Banglish) common।
    • 
          
  • Cultural concept — local clothing, food, festival।
    • 
        

        
Performance assessment:

        
    
          
  • Bangla zero-shot — 50-65% on standard benchmarks (vs 75%+ English)।
    • 
          
  • Bangladesh-specific (rickshaw, jamdani) — even lower।
    • 
          
  • Improvement actively researched।
    • 
        

        
Bangladesh strategies:

        
    
          
  • Use M-CLIP base: immediate availability।
    • 
          
  • Translate + English CLIP: Bangla → English → CLIP। Translation quality critical।
    • 
          
  • Finetune on local data: few thousand Bangla image-caption pair। CLIP base + LoRA finetune।
    • 
          
  • Synthetic data: existing Bangla NLP corpus + image generation।
    • 
        

        
Local data sources:

        
    
          
  • Bangladesh Wikipedia।
    • 
          
  • Prothom Alo, Daily Star image captions।
    • 
          
  • Bornom OCR, BUET datasets।
    • 
          
  • Crowdsourced caption।
    • 
        

        
Application:

        
    
          
  • Bangla image search engine।
    • 
          
  • Local content moderation।
    • 
          
  • Educational tool (student question — find image)।
    • 
          
  • Cultural digitization।
    • 
        

        
Cost reality:

        
    
          
  • Pretrain — multi-million dollar (impossible local)।
    • 
          
  • Finetune — $100-1000 (feasible)।
    • 
          
  • API use — pay-per-call।
    • 
        

        
মূল উপলব্ধি: Bangla CLIP — emerging field। Foundation model ecosystem leverage critical। BUET, BUP, NSU research groups — focus area।

      

    


    

      
        প্র ০৪
        Stable Diffusion-এ CLIP key role। Generation-এ কীভাবে use? Without CLIP, image generation কেমন হত?
      
      

        
CLIP — Stable Diffusion-এর "language understanding" component।

        
Role in generation:

        
    
          
  • User prompt: "A cat sitting on a moon"।
    • 
          
  • CLIP text encoder → 768-D embedding।
    • 
          
  • U-Net cross-attention input।
    • 
          
  • Each diffusion step — text guides generation।
    • 
        

        
Why CLIP encoder:

        
    
          
  • Image-text aligned → effectively "what would this prompt look like"।
    • 
          
  • Rich semantic — color, object, action capture।
    • 
          
  • Pre-trained — generation training save।
    • 
        

        
Without CLIP:

        
    
          
  • Pre-CLIP generation: GAN with class label (ImageNet)।
    • 
          
  • BigGAN — 1000 class। Unable to generate "purple elephant"।
    • 
          
  • Pixel-wise text-to-image (DALL-E 1) — autoregressive, slow।
    • 
        

        
Stable Diffusion-এ specifically:

        
    
          
  • Original SD: CLIP ViT-L/14।
    • 
          
  • SD 2: OpenCLIP ViT-H/14।
    • 
          
  • SD-XL: dual encoder (CLIP-L + OpenCLIP-G)।
    • 
          
  • SD 3: T5 encoder + CLIP।
    • 
          
  • FLUX: T5 + CLIP।
    • 
        

        
T5 vs CLIP:

        
    
          
  • T5 — pure text encoder, no vision alignment।
    • 
          
  • Better long prompt understanding।
    • 
          
  • Compositional reasoning improved।
    • 
          
  • Modern SD use both।
    • 
        

        
DALL-E 3:

        
    
          
  • OpenAI internal architecture।
    • 
          
  • GPT-4 likely involved in prompt understanding।
    • 
          
  • Best compositional generation।
    • 
        

        
Bangladeshi creative AI:

        
    
          
  • Bangla prompt translate → English → SD।
    • 
          
  • SD finetune on Bangladesh visual culture।
    • 
          
  • LoRA — Bengali New Year, jamdani style।
    • 
        

        
Future direction:

        
    
          
  • Native multimodal generation — Sora, Gemini।
    • 
          
  • Single model both text + image native।
    • 
          
  • CLIP-style separate encoder eventually phase out।
    • 
        

        
মূল উপলব্ধি: CLIP → diffusion model bridge। Without "understanding" component, generation quality limit। Multimodal foundation evolution।

      

    

  


  

    
 অনুশীলন

    
    
      
  1. 
        Zero-shot classify: CLIP দিয়ে একটি image-এ ৫ class candidate-এর softmax probability।
        
    
          
          
    
            
    Code section ৬-এ। 5 class change করে test।
    
          
    
        
    
      
    2. 

      
  2. 
        Similarity matrix: ১০ image, ১০ text caption — pairwise cosine similarity। Heatmap visualize।
        
    
          
          
    
            
    img_emb = model.encode_image(images)  # (10, 512)
txt_emb = model.encode_text(texts)
sim = (img_emb / img_emb.norm(dim=-1, keepdim=True)) @ (txt_emb / txt_emb.norm(dim=-1, keepdim=True)).T
# matplotlib heatmap
    
          
    
        
    
      
    3. 

      
  3. 
        ভাবুন: Bangladesh news image archive search engine — CLIP কীভাবে অপ্রতিরোধ্য?
        
    
          
          
    
            
    (1) Index all image with CLIP embedding stored। (2) User Bangla query → translate to English (or M-CLIP)। (3) Cosine similarity search top-k। Daily Star, Prothom Alo style application।
    
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন · ABCL TECH-এ আপনার পরবর্তী পদক্ষেপ

    
    

      কোড রানার কাজ না করলে? Google Colab use করুন।
    

  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ২৬ · Swin Transformer