পাঠ ২৭ · ৩৫-এর মধ্যে · মডিউল ৪
Home / AI Courses / Computer Vision / CLIP

CLIP — vision-language

CLIP — Contrastive Language–Image Pretraining
৮ মিনিট পড়া উচ্চ · Advanced Python কোডসহ

এই পাঠে যা শিখবেন

  • CLIP architecture — dual encoder
  • Contrastive learning (InfoNCE loss)
  • Zero-shot classification
  • Multimodal applications

১ · Vision-language alignment

Traditional CV — fixed class set (ImageNet 1000). Real world — open vocabulary. "What's in this image?" answer infinite.

CLIP-এর insight: natural language-ই caption supervision. Web-এ billions of image-text pair available.

কেন্দ্রীয় ধারণা

CLIP = "image embedding ও text embedding-কে align করা"। Same embedding space-এ image ও তার description close, mismatched far. Zero-shot transfer enabled.

২ · CLIP architecture

(ক) Image encoder

  • ResNet বা ViT.
  • Image → feature vector (512-D for ViT-B).
  • Linear projection → shared embedding space.

(খ) Text encoder

  • Transformer (GPT-style).
  • Text → token embedding → final token vector (512-D).
  • Linear projection → shared embedding space.

(গ) Shared 512-D space

  • Image embedding ও text embedding একই dimension.
  • Cosine similarity compare করা যায়।

৩ · Contrastive learning

Training-এ batch (e.g., 32K) image-text pair. Aim: matched pair close, mismatched far.

InfoNCE loss:

$$\mathcal{L} = -\frac{1}{N} \sum_i \log \frac{\exp(\text{sim}(I_i, T_i) / \tau)}{\sum_j \exp(\text{sim}(I_i, T_j) / \tau)}$$

  • $\text{sim}$ — cosine similarity.
  • $\tau$ — temperature (learnable).
  • Positive: $(I_i, T_i)$ matched.
  • Negative: $(I_i, T_{j \ne i})$ — N-1 mismatched.
  • Symmetric loss — text-to-image direction also.

৪ · Training data

  • WIT (Web Image Text) — 400M image-text pair.
  • Crawled internet — caption, alt-text, etc.
  • Diverse — open vocabulary capture.
  • Noisy — কিন্তু scale-এ overcome.

৫ · Zero-shot classification

No fine-tune. Just rephrase classification as caption matching:

  1. Class names: ["cat", "dog", "car"].
  2. Build caption: "A photo of a cat", "A photo of a dog", ...।
  3. Encode all caption — text embeddings.
  4. Encode image — image embedding.
  5. Cosine similarity image vs each caption.
  6. Max similarity → predicted class.

Result: ImageNet zero-shot — 76.2% top-1 (CLIP ViT-L). Without ImageNet training!

Pre-CLIP: "I trained on cats and dogs only — anything else, I don't know". CLIP: "I learnt language-image relationship — describe anything in words, I can find it visually".
CLIP — dual encoder contrastive learning 📷 Image cat photo Image Encoder ViT-L (300M) image emb 512-D 📝 Text "a photo of a cat" Text Encoder Transformer (63M) text emb 512-D Cosine sim → matched/not Training: matched (cat ↔ "cat photo") high, mismatched low InfoNCE loss — softmax over batch negatives 400M image-text pair, web-scraped Zero-shot transfer to any classification by text prompt
CLIP — image ও text encoder একই embedding space-এ। Contrastive loss দিয়ে align.

৬ · CLIP use

Python · CLIP 
# pip install git+https://github.com/openai/CLIP.git
import clip
import torch
from PIL import Image

device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-L/14", device=device)

# Image preprocess
img = preprocess(Image.open("photo.jpg")).unsqueeze(0).to(device)

# Text candidates
texts = clip.tokenize([
    "a photo of a cat",
    "a photo of a dog",
    "a photo of a car"
]).to(device)

# Compute features
with torch.no_grad():
    image_features = model.encode_image(img)
    text_features = model.encode_text(texts)

# Cosine similarity
image_features = image_features / image_features.norm(dim=-1, keepdim=True)
text_features = text_features / text_features.norm(dim=-1, keepdim=True)
similarity = (image_features @ text_features.T).softmax(dim=-1)

print("Probabilities:", similarity[0].tolist())
# e.g., cat=0.92, dog=0.06, car=0.02

    


    

      Zero-shot — no training, no labels. Just text descriptions. CLIP understanding language-image association.
    


    
৭ · CLIP applications

    
    
      
  • Zero-shot classification: any class via text prompt.
    • 
      
  • Image search: "find sunset over mountain" → matching image.
    • 
      
  • Stable Diffusion: text condition encoder.
    • 
      
  • DALL-E: generation conditioning.
    • 
      
  • Open-vocabulary detection: Grounding DINO, OWL-ViT.
    • 
      
  • Image captioning: CLIP feature → LM generate caption.
    • 
      
  • Content moderation: "adult content" probability.
    • 
    


    
৮ · Multilingual CLIP

    
    
      
  • M-CLIP: 50+ languages. Bangla supported.
    • 
      
  • OpenCLIP: open implementation, multilingual variant.
    • 
      
  • SigLIP (২০২৩): sigmoid loss replace softmax — multilingual better.
    • 
      
  • BLIP, BLIP-2: generative + understanding.
    • 
    


    
৯ · CLIP limitations

    
    
      
  • Counting weakness: "3 cats" — CLIP struggles.
    • 
      
  • Spatial reasoning: "cat to left of dog" — limited.
    • 
      
  • Bias: web-scraped data — gender/race bias.
    • 
      
  • Fine-grained: bird species — coarse only.
    • 
      
  • Compositional: "red cube on blue sphere" — wrong sometimes.
    • 
    


    
১০ · CLIP variants

    
    
      
  • ALIGN (Google): 1B noisy image-text pair.
    • 
      
  • BASIC, FILIP: token-level alignment.
    • 
      
  • SigLIP: sigmoid loss.
    • 
      
  • EVA-CLIP: EVA backbone.
    • 
      
  • BLIP-2: Q-Former — better captioning.
    • 
    


    

      CLIP-এর success-এর key — scale + diverse data. Bangladesh-এ training cost prohibitive — pretrained CLIP/SigLIP use. Bangla specific finetune সম্ভব ছোট cost-এ।
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      
        প্র ০১
        "A photo of a cat" — কেন এই specific prompt? "Cat" alone কাজ করে কিন্তু কম accurate. Prompt engineering CV-তে?
      
      

        
CLIP-এর underrated detail — prompt template choice. "Prompt engineering" CV-তেও matters.

        
Why "A photo of":

        
    
          
  • Training data — captions look like "a photo of X".
    • 
          
  • Model learnt this pattern.
    • 
          
  • Distribution match.
    • 
        

        
Effect on accuracy:

        
    
          
  • "cat" — 70%.
    • 
          
  • "a photo of a cat" — 76%.
    • 
          
  • "a photo of a {label}, a type of {category}" — 78%.
    • 
          
  • +8% from prompt alone!
    • 
        

        
Prompt ensembling:

        
    
          
  • Multiple prompt: "a photo of {}", "a picture of {}", "an image of {}"।
    • 
          
  • Average embeddings.
    • 
          
  • Robust.
    • 
        

        
Domain-specific:

        
    
          
  • Medical: "a chest X-ray showing {}"।
    • 
          
  • Satellite: "a satellite photo of {}"।
    • 
          
  • Drawing: "a sketch of {}"।
    • 
        

        
Bangla challenge:

        
    
          
  • OpenCLIP Bangla support — "একটি {} ছবি"।
    • 
          
  • Mixed: "a photo of {} (a {})" — English category + Bangla.
    • 
          
  • Test multiple, see what works.
    • 
        

        
Auto prompt:

        
    
          
  • CoOp (২০২২): learnable prompt token.
    • 
          
  • CoCoOp: conditional on image.
    • 
          
  • Adapt to dataset-specific.
    • 
        

        
মূল উপলব্ধি: Prompt = vital interface. Multimodal model-এ language-CV bridge careful design.

      

    


    

      
        প্র ০২
        CLIP zero-shot ImageNet 76% — supervised-এর কাছাকাছি। কীভাবে natural language supervision-এ এত strong?
      
      

        
CLIP-এর core insight — language label-এর চেয়ে rich.

        
ImageNet label:

        
    
          
  • 1 word/short phrase per image.
    • 
          
  • Discrete category.
    • 
          
  • 1000 class fixed.
    • 
        

        
Web caption:

        
    
          
  • Variable length sentence.
    • 
          
  • Continuous natural language.
    • 
          
  • Open vocabulary.
    • 
          
  • Compositional (color + object + action).
    • 
        

        
Information density:

        
    
          
  • Caption "fluffy orange tabby cat sitting on red carpet" — much more signal than "cat".
    • 
          
  • Multi-label implicit.
    • 
          
  • Spatial, attribute, context all encode.
    • 
        

        
Scale advantage:

        
    
          
  • ImageNet 1.4M label-image pair.
    • 
          
  • CLIP 400M image-text pair.
    • 
          
  • 300x more data + richer label.
    • 
        

        
Generalization:

        
    
          
  • Web-scraped — natural distribution.
    • 
          
  • Long-tail visual concept.
    • 
          
  • Adapts to new task without retraining.
    • 
        

        
Empirical:

        
    
          
  • CLIP linear probe ImageNet — 86%.
    • 
          
  • Above ResNet supervised on ImageNet itself.
    • 
          
  • Out-of-distribution robustness — far better.
    • 
        

        
Cost:

        
    
          
  • CLIP train — multi-million dollar GPU.
    • 
          
  • One time investment, broad applicability.
    • 
          
  • OpenAI-Anthropic-Google cost-affordable.
    • 
        

        
মূল উপলব্ধি: Label-এর nature matters. "Natural language as supervision" — paradigm shift.

      

    


    

      
        প্র ০৩
        Bangla CLIP — Bangladesh data-এর জন্য কী option? OpenCLIP মতো resources-এ Bangla আছে?
      
      

        
Multilingual CLIP — Bangla support gradually improving.

        
Available options:

        
    
          
  • M-CLIP (Multilingual CLIP): 50+ language, Bangla included. OpenAI CLIP-এর extension.
    • 
          
  • OpenCLIP: Apache 2.0 license, open weight. Multilingual variants.
    • 
          
  • SigLIP: Google, multilingual better generally.
    • 
          
  • NLLB-CLIP: Meta, 200+ language including Bangla.
    • 
        

        
Bangla-specific challenges:

        
    
          
  • Web data — Bangla less crawled than English.
    • 
          
  • Caption quality variable.
    • 
          
  • Code-mixed (Banglish) common.
    • 
          
  • Cultural concept — local clothing, food, festival.
    • 
        

        
Performance assessment:

        
    
          
  • Bangla zero-shot — 50-65% on standard benchmarks (vs 75%+ English).
    • 
          
  • Bangladesh-specific (rickshaw, jamdani) — even lower.
    • 
          
  • Improvement actively researched.
    • 
        

        
Bangladesh strategies:

        
    
          
  • Use M-CLIP base: immediate availability.
    • 
          
  • Translate + English CLIP: Bangla → English → CLIP. Translation quality critical.
    • 
          
  • Finetune on local data: few thousand Bangla image-caption pair. CLIP base + LoRA finetune.
    • 
          
  • Synthetic data: existing Bangla NLP corpus + image generation.
    • 
        

        
Local data sources:

        
    
          
  • Bangladesh Wikipedia.
    • 
          
  • Prothom Alo, Daily Star image captions.
    • 
          
  • Bornom OCR, BUET datasets.
    • 
          
  • Crowdsourced caption.
    • 
        

        
Application:

        
    
          
  • Bangla image search engine.
    • 
          
  • Local content moderation.
    • 
          
  • Educational tool (student question — find image).
    • 
          
  • Cultural digitization.
    • 
        

        
Cost reality:

        
    
          
  • Pretrain — multi-million dollar (impossible local).
    • 
          
  • Finetune — $100-1000 (feasible).
    • 
          
  • API use — pay-per-call.
    • 
        

        
মূল উপলব্ধি: Bangla CLIP — emerging field. Foundation model ecosystem leverage critical. BUET, BUP, NSU research groups — focus area.

      

    


    

      
        প্র ০৪
        Stable Diffusion-এ CLIP key role. Generation-এ কীভাবে use? Without CLIP, image generation কেমন হত?
      
      

        
CLIP — Stable Diffusion-এর "language understanding" component.

        
Role in generation:

        
    
          
  • User prompt: "A cat sitting on a moon".
    • 
          
  • CLIP text encoder → 768-D embedding.
    • 
          
  • U-Net cross-attention input.
    • 
          
  • Each diffusion step — text guides generation.
    • 
        

        
Why CLIP encoder:

        
    
          
  • Image-text aligned → effectively "what would this prompt look like".
    • 
          
  • Rich semantic — color, object, action capture.
    • 
          
  • Pre-trained — generation training save.
    • 
        

        
Without CLIP:

        
    
          
  • Pre-CLIP generation: GAN with class label (ImageNet).
    • 
          
  • BigGAN — 1000 class. Unable to generate "purple elephant".
    • 
          
  • Pixel-wise text-to-image (DALL-E 1) — autoregressive, slow.
    • 
        

        
Stable Diffusion-এ specifically:

        
    
          
  • Original SD: CLIP ViT-L/14.
    • 
          
  • SD 2: OpenCLIP ViT-H/14.
    • 
          
  • SD-XL: dual encoder (CLIP-L + OpenCLIP-G).
    • 
          
  • SD 3: T5 encoder + CLIP.
    • 
          
  • FLUX: T5 + CLIP.
    • 
        

        
T5 vs CLIP:

        
    
          
  • T5 — pure text encoder, no vision alignment.
    • 
          
  • Better long prompt understanding.
    • 
          
  • Compositional reasoning improved.
    • 
          
  • Modern SD use both.
    • 
        

        
DALL-E 3:

        
    
          
  • OpenAI internal architecture.
    • 
          
  • GPT-4 likely involved in prompt understanding.
    • 
          
  • Best compositional generation.
    • 
        

        
Bangladeshi creative AI:

        
    
          
  • Bangla prompt translate → English → SD.
    • 
          
  • SD finetune on Bangladesh visual culture.
    • 
          
  • LoRA — Bengali New Year, jamdani style.
    • 
        

        
Future direction:

        
    
          
  • Native multimodal generation — Sora, Gemini.
    • 
          
  • Single model both text + image native.
    • 
          
  • CLIP-style separate encoder eventually phase out.
    • 
        

        
মূল উপলব্ধি: CLIP → diffusion model bridge. Without "understanding" component, generation quality limit. Multimodal foundation evolution.

      

    

  


  

    
 অনুশীলন

    
    
      
  1. 
        Zero-shot classify: CLIP দিয়ে একটি image-এ ৫ class candidate-এর softmax probability.
        
    
          
          
    
            
    Code section ৬-এ। 5 class change করে test.
    
          
    
        
    
      
    2. 

      
  2. 
        Similarity matrix: ১০ image, ১০ text caption — pairwise cosine similarity. Heatmap visualize.
        
    
          
          
    
            
    img_emb = model.encode_image(images)  # (10, 512)
txt_emb = model.encode_text(texts)
sim = (img_emb / img_emb.norm(dim=-1, keepdim=True)) @ (txt_emb / txt_emb.norm(dim=-1, keepdim=True)).T
# matplotlib heatmap
    
          
    
        
    
      
    3. 

      
  3. 
        ভাবুন: Bangladesh news image archive search engine — CLIP কীভাবে অপ্রতিরোধ্য?
        
    
          
          
    
            
    (1) Index all image with CLIP embedding stored. (2) User Bangla query → translate to English (or M-CLIP). (3) Cosine similarity search top-k. Daily Star, Prothom Alo style application.
    
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন · ABCL TECH-এ আপনার পরবর্তী পদক্ষেপ

    
    

      কোড রানার কাজ না করলে? Google Colab use করুন।
    

  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ২৬ · Swin Transformer