পাঠ ২৫ · ২৮-এর মধ্যে · মডিউল ৪
Home / AI Courses / Generative AI / Evaluation

Evaluation — FID, CLIP score

Evaluation metrics — FID, IS, CLIP score, human eval
৬ মিনিট পড়া মধ্যবর্তী · Intermediate Metrics

এই পাঠে যা শিখবেন

  • FID-র math ও কেন এটি dominant generative metric
  • IS, CLIP score, LPIPS — কোন metric কখন
  • Each metric-এর limitation
  • Human evaluation কীভাবে design করতে হয়

১ · কেন evaluation কঠিন

Discriminative task-এ accuracy clear — predict ↔ ground-truth। Generative task-এ "ground truth" নেই — multiple valid output। "এই AI-generated image কত ভাল" — উত্তর subjective।

চাই এমন metric যা:

  • Quality: photorealism / coherence।
  • Diversity: mode collapse detect।
  • Fidelity: text prompt-এর সাথে match।
  • Calibrated: human judgment correlate।

২ · FID — Fréchet Inception Distance (Heusel 2017)

Most popular generative metric। Idea:

  1. Real image set ও generated image set — দু'টাই।
  2. প্রতিটি Inception V3 (ImageNet pretrained)-এর penultimate layer-এ pass — ২০৪৮-D feature।
  3. Real features → multivariate Gaussian $\mathcal{N}(\mu_r, \Sigma_r)$।
  4. Generated features → $\mathcal{N}(\mu_g, \Sigma_g)$।
  5. Fréchet distanceFréchet distance (Wasserstein-2)দু'টি Gaussian distribution-এর মধ্যে দূরত্বের মাপ। মেট্রিক — FID-এর "দূরত্ব" component। দু'টি Gaussian-এর মধ্যে:

$$\text{FID} = \|\mu_r - \mu_g\|^2 + \text{tr}\left(\Sigma_r + \Sigma_g - 2(\Sigma_r \Sigma_g)^{1/2}\right)$$

FID interpretation

FID < ১০: photorealistic। FID ১০-৩০: clearly AI। FID > ৫০: poor। SOTA text-to-image (SD3, FLUX) MS-COCO-তে FID ~৭-১০।

৩ · Inception Score (Salimans 2016) — older but historical

Idea: ভাল generated image-এ —

  • Conditional class probability $p(y|x)$ peaked (clear class)।
  • Marginal $p(y) = \mathbb{E}_x p(y|x)$ uniform (diverse)।

$$\text{IS} = \exp\left(\mathbb{E}_x \, \text{KL}(p(y|x) \,\|\, p(y))\right)$$

বেশি ভাল। কিন্তু — ImageNet-trained classifier-নির্ভর; mode collapse detect না; real reference ব্যবহার করে না। FID-এর কাছে hara।

৪ · CLIP score — text-to-image fidelity

Text-to-image-এ FID alone যথেষ্ট নয় — image ভাল হলেও prompt-এর সাথে মিলবে এমন গ্যারান্টি নেই। CLIP score:

$$\text{CLIPScore}(I, T) = \max(0, \cos(\text{CLIP-image}(I), \text{CLIP-text}(T)))$$

০.২৫-০.৪ = excellent alignment। ০.১৫-০.২৫ = decent। < ০.১ = misaligned।

৫ · LPIPS, DreamSim — perceptual similarity

  • LPIPS (Zhang 2018): AlexNet/VGG feature-এ distance। Pixel-MSE-এর চেয়ে human-aligned।
  • DreamSim (২০২৩): DINO feature + human-trained alignment। Modern best similarity।
  • SSIM, PSNR: pixel-level — reconstruction-এ। Generation-এ কম প্রাসঙ্গিক।

৬ · Human evaluation — gold standard

  • Side-by-side preference: "A vs B — কোনটা better?" Bradley-Terry model দিয়ে rank।
  • Likert scale: 1-5 rating।
  • Aspect-specific: photorealism, prompt alignment, aesthetic — পৃথক।
  • Adversarial: "এটা AI না real?" — ৫০% accuracy = indistinguishable।
  • Crowd platform: MTurk, Prolific, Scale AI।
  • Expert eval: domain-specific (medical, legal)।
FID — দু'টি distribution-এর মধ্যে দূরত্ব 📷 Real images N=10K-50K samples Inception V3 penultimate layer (2048-D) N(μ_r, Σ_r) real Gaussian 🤖 Generated images N samples from model Inception V3 same encoder N(μ_g, Σ_g) generated Gaussian 📏 FID ||μ_r − μ_g||² + tr(Σ_r+Σ_g−2(Σ_rΣ_g)^½) lower = better SOTA text-to-image FID on MS-COCO ≈ 7-10
FID — Inception feature-এ real ও generated-কে Gaussian model করে দু'য়ের Fréchet distance।

৭ · FID compute — torchmetrics

Python · torchmetrics 
# pip install torchmetrics torch-fidelity
from torchmetrics.image.fid import FrechetInceptionDistance
import torch

fid = FrechetInceptionDistance(feature=2048, normalize=True).cuda()

# Real images batch (uint8 [0,255] or float [0,1] with normalize=True)
real = torch.rand(100, 3, 299, 299).cuda()  # placeholder
gen  = torch.rand(100, 3, 299, 299).cuda()  # placeholder

fid.update(real, real=True)
fid.update(gen,  real=False)
print(f"FID: {fid.compute().item():.2f}")

    


    
৮ · CLIP score compute


    

      

        
        Python · CLIPScore
      

      
from torchmetrics.multimodal.clip_score import CLIPScore

clip = CLIPScore(model_name_or_path="openai/clip-vit-base-patch16").cuda()

prompts = ["a colorful Bangladeshi rickshaw painting style",
           "rice paddy fields at sunset"]
images  = torch.randint(0, 255, (2, 3, 224, 224)).cuda()

score = clip(images, prompts)
print(f"CLIP score: {score.item():.3f}")

      

        
        
      

      

    


    

      Same metric ভিন্ন implementation-এ ভিন্ন value দেয় — Inception V3 weight version, image preprocessing, batch size সব impact। Paper-এ "CleanFID" library-র recommendation।
    


    

      Modern benchmark suite: HEIM (Stanford), VBench (video), GenAI-Bench — multi-aspect evaluation।
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      
        প্র ০১
        FID এত popular — কিন্তু ২০২৩-২৫-এ critique বাড়ছে। FID-র limitations কী? Replacement কী?
      
      

        
FID ৮ বছর-এর reigning king। কিন্তু modern generative model-এর scale-এ cracks visible।

        
FID-র assumption:

        
    
          
  • Inception feature distribution Gaussian — reality-এ নয়।
    • 
          
  • Inception ImageNet-trained — natural object-bias। Art, satellite, medical-এ less suitable।
    • 
          
  • Sample size sensitive — < ১০K-এ noisy।
    • 
          
  • Single feature space — multi-aspect quality miss।
    • 
        

        
Documented failure cases:

        
    
          
  • Stein et al. 2023 (NeurIPS): "Exposing flaws of generative model evaluation metrics"। FID human preference correlation poor at high quality।
    • 
          
  • Memorization-blind: exact training image regenerate → FID excellent but generation invalid।
    • 
          
  • Diversity vs quality tradeoff: FID hide।
    • 
          
  • Compositional failures: "blue cube on red sphere" wrong → FID fine।
    • 
          
  • Inception fails on text/UI: ImageNet category-এ text/UI absent।
    • 
        

        
Replacements:

        
    
          
  • FD-DINOv2 (Stein 2023): Inception-এর বদলে DINOv2 — self-supervised, better।
    • 
          
  • CMMD (Jayasumana 2024, Google): CLIP feature + maximum mean discrepancy। Sample-efficient (<১K)।
    • 
          
  • Precision-Recall (Kynkäänniemi 2019): precision = quality, recall = diversity — separate measure।
    • 
          
  • Density-Coverage (Naeem 2020): Precision-Recall-এর improvement।
    • 
          
  • Memorization rate: retrieval similarity-এ near-duplicate count।
    • 
          
  • HPSv2, ImageReward: human preference fitted reward model।
    • 
        

        
Modern best practice:

        
    
          
  • Multiple metrics report — FID + CLIP score + HPS + sample diversity।
    • 
          
  • Specific aspect — text alignment, aesthetic, artifact-free।
    • 
          
  • Domain-specific — fashion-FID, satellite-FID।
    • 
          
  • Human eval crucial decisions।
    • 
          
  • Memorization audit — retrieve nearest training image।
    • 
        

        
Research trend:

        
    
          
  • Multimodal LLM as judge (Gemini, GPT-4) — fast & flexible।
    • 
          
  • Reference-free quality (No-reference)।
    • 
          
  • Compositional benchmarks (Compbench, T2I-CompBench)।
    • 
          
  • Bias auditing — demographic balance।
    • 
        

        
মূল উপলব্ধি: No metric perfect। FID still useful baseline, কিন্তু basket-of-metrics + human eval সর্বদা উত্তম। "Optimize for metric" ≠ "optimize for users"।

      

    


    

      
        প্র ০২
        একটি বাংলা poster generator-এ আপনি কোন metric ব্যবহার করবেন? FID Bangladeshi context-এ কাজ করে কি?
      
      

        
Bangladeshi context-এ standard metric-ই insufficient — multiple reason।

        
FID-র Bangla problem:

        
    
          
  • Inception V3 ImageNet-এ trained — Bangladeshi visual culture absent।
    • 
          
  • Saree, lungi, panjabi misclassify।
    • 
          
  • Rickshaw, CNG, nouka — no class।
    • 
          
  • Reference set কোথায়? "Real Bangladeshi image" curated dataset rare।
    • 
        

        
CLIP score Bangla problem:

        
    
          
  • OpenAI CLIP English-only। Bangla prompt → tokenizer fail।
    • 
          
  • Multilingual CLIP available কিন্তু Bangla quality moderate।
    • 
          
  • Cultural concept — "ঈদ", "পুজা" weak embedding।
    • 
        

        
Practical metric stack for Bangla poster:

        
    
          
  • 
            Text rendering accuracy: generated image-এ Bangla text correctly written কি? OCR (Google Vision Bangla) → match prompt text।
          
    • 
          
  • 
            Cultural CLIP score: Multilingual CLIP (XLM-R-CLIP) দিয়ে।
          
    • 
          
  • 
            FD-DINOv2: Inception-এর বদলে DINOv2 — better non-natural domain।
          
    • 
          
  • 
            Aesthetic score: LAION aesthetic predictor।
          
    • 
          
  • 
            Object presence: YOLO Bangla custom-trained — rickshaw, sari detect।
          
    • 
          
  • 
            Color palette: traditional Bangla color (Boishakh red, ঈদ green) histogram match।
          
    • 
          
  • 
            Human eval (Bangladeshi reviewer): ১০ poster pair side-by-side।
          
    • 
        

        
Reference dataset to curate:

        
    
          
  • Prothom Alo poster archive।
    • 
          
  • Bangladesh tourism board ছবি।
    • 
          
  • Old Bangla movie poster।
    • 
          
  • Government poster (election, DGHS)।
    • 
          
  • Cultural festival photo।
    • 
          
  • ~১০K curated reference enough for FD-DINOv2।
    • 
        

        
Bangla-specific failure mode:

        
    
          
  • Bangla glyphs garbled — most diffusion model can't render।
    • 
          
  • Hand pose unnatural in lungi-saree।
    • 
          
  • Rickshaw — wheel/structure wrong।
    • 
          
  • Mosque architecture — Western church confusion।
    • 
        

        
Annotation guideline (Human eval):

        
    
          
  • Native Bangla speaker, mid-age।
    • 
          
  • Aspect — cultural authenticity, text legibility, aesthetic, prompt match।
    • 
          
  • Pairwise > scale (less bias)।
    • 
          
  • ৫০-১০০ pair × ৫ rater minimum।
    • 
        

        
Tool stack:

        
    
          
  • torchmetrics FID, CLIP score।
    • 
          
  • cleanfid reproducible।
    • 
          
  • Custom OCR-pass-rate Python script।
    • 
          
  • Argilla / Label Studio human eval UI।
    • 
          
  • WandB / MLflow tracking।
    • 
        

        
মূল উপলব্ধি: Standard metric Bangla-blind। Domain-aware curation + human eval — গুরুত্বপূর্ণ। বাংলাদেশী context-এ কাজ করতে হলে নিজের benchmark বানানো দরকার।

      

    


    

      
        প্র ০৩
        "LLM-as-judge" — GPT-4 দিয়ে generated content evaluate। Pros, cons, bias? Cheap human eval-এর alternative কি?
      
      

        
২০২৩-এ MT-Bench, AlpacaEval — "GPT-4 judge" mainstream হয়। দ্রুত ও cheap কিন্তু complications।

        
Pros:

        
    
          
  • $0.01-0.10/eval — human-এর ১০০x cheap।
    • 
          
  • Scale — ১০K eval/hour।
    • 
          
  • Reproducible (with seed)।
    • 
          
  • Multi-aspect evaluation in one call।
    • 
          
  • Multilingual — Bangla judge সম্ভব।
    • 
        

        
Cons & biases:

        
    
          
  • Position bias: "A vs B"-এ first option judge favor। Mitigation: swap test।
    • 
          
  • Length bias: longer answer = better judged। Wrong।
    • 
          
  • Style bias: formal, structured response over substantive but informal।
    • 
          
  • Self-bias: GPT-4 GPT-4 output other model output-এর চেয়ে ভাল judge করে।
    • 
          
  • Refusal bias: safety-trained model conservative judge।
    • 
          
  • Sycophancy: "the user wanted this answer" — alignment dependent।
    • 
          
  • Domain blindness: medical, legal, niche — superficial judgment।
    • 
        

        
Mitigation strategies:

        
    
          
  • Pairwise + swap: A vs B, then B vs A → average।
    • 
          
  • Chain-of-thought judge: reason before verdict।
    • 
          
  • Multi-judge ensemble: GPT-4 + Claude + Gemini → majority vote।
    • 
          
  • Reference-based: ground truth provided → reduce ambiguity।
    • 
          
  • Rubric anchoring: specific criteria, scale anchor description।
    • 
          
  • Human spot-check: ৫-১০% random human verify।
    • 
        

        
When to use LLM judge:

        
    
          
  • ✅ Iteration speed — model dev cycle।
    • 
          
  • ✅ A/B regression test।
    • 
          
  • ✅ Multilingual eval at scale।
    • 
          
  • ✅ Subjective quality (writing, summarization)।
    • 
        

        
When NOT:

        
    
          
  • ❌ Final benchmark publication — use human।
    • 
          
  • ❌ Safety-critical (medical, legal) — expert needed।
    • 
          
  • ❌ Self-evaluation (own model) — biased।
    • 
          
  • ❌ Tasks requiring expertise judge lacks।
    • 
        

        
Hybrid recommendation:

        
    
          
  • Daily dev: LLM-as-judge full automation।
    • 
          
  • Weekly: ১০% human spot-check।
    • 
          
  • Monthly / release: full human eval ১০০-৩০০ sample।
    • 
          
  • Public benchmark: human-only।
    • 
        

        
Tools:

        
    
          
  • lm-evaluation-harness, g-eval, DeepEval, RAGAS।
    • 
          
  • Anthropic Claude judge for OpenAI; vice versa — reduce self-bias।
    • 
          
  • Open-source judge — Prometheus 2, JudgeLM।
    • 
        

        
Modern direction:

        
    
          
  • Reward model (PairRM, ArmoRM) judge-এর চেয়ে calibrated।
    • 
          
  • Fine-tuned Bangla judge — community দরকার।
    • 
          
  • Verifier model (mathematical, code) — ground truth available।
    • 
        

        
মূল উপলব্ধি: LLM judge = "fast cheap proxy"। Human eval-এর replacement নয়, complement। Pipeline-এ wisely place করুন।

      

    


    

      
        প্র ০৪
        Memorization audit — generative model কখনো training image verbatim copy করে কি? Detect কীভাবে? Copyright lawsuit-এ এই evidence কতটা গুরুত্বপূর্ণ?
      
      

        
NYT vs OpenAI, Getty vs Stability, Authors Guild vs OpenAI — সব lawsuit-এ memorization key evidence।

        
Memorization happen কেন:

        
    
          
  • Repeat data (popular image, viral text) over-represented।
    • 
          
  • Small dataset overfitting।
    • 
          
  • Long training dilute সংস্করণ।
    • 
          
  • Verbatim memorization rare (~০.০১%) but real।
    • 
        

        
Detection methods:

        
    
          
  • 
            Carlini et al. 2023 (USENIX): Stable Diffusion ১০৯টি training image verbatim recover। Method: prompt-এ training caption + many seeds → near-duplicate retrieve।
          
    • 
          
  • 
            Membership inference: এই image training-এ ছিল কি? Loss-pattern-এ inference।
          
    • 
          
  • 
            Embedding nearest-neighbor: generated image-এর CLIP embedding-এ training set-এ closest distance।
          
    • 
          
  • 
            Pixel-level diff: SSIM, LPIPS threshold।
          
    • 
          
  • 
            Attribution attacks: "find a training image very similar to this output"।
          
    • 
        

        
For LLM:

        
    
          
  • Carlini 2021 — GPT-2 PII, code verbatim।
    • 
          
  • NYT lawsuit — GPT-4 prompted continue article → near-verbatim।
    • 
          
  • "Extractable memorization" — special prompt trigger।
    • 
        

        
Legal landscape:

        
    
          
  • NYT v OpenAI: evidence GPT-4 NYT articles continue। OpenAI defense — "fair use, training is transformative"।
    • 
          
  • Getty v Stability: Stable Diffusion output-এ Getty watermark visible — direct memorization evidence।
    • 
          
  • Andersen et al. v Stability: artist-style replication — different from verbatim, harder to prove।
    • 
          
  • EU AI Act: training data summary disclosure।
    • 
          
  • Japan / Singapore: AI training fair-use friendly।
    • 
        

        
Mitigation (model dev side):

        
    
          
  • Deduplication: training data near-duplicate remove।
    • 
          
  • Differential privacy: noise inject — verbatim hard।
    • 
          
  • Output filtering: generated-এ training nearest-neighbor check।
    • 
          
  • Concept erasure (TIME, ESD): specific concept "unlearn"।
    • 
          
  • Provenance log: contribution per training example।
    • 
        

        
Bangladesh angle:

        
    
          
  • Bangla literature, Tagore — public domain বনাম private estate।
    • 
          
  • News organizations (Prothom Alo, BBC Bangla) — same NYT-style claim।
    • 
          
  • Local artists, photographers — protection mechanism নেই।
    • 
          
  • Bangladesh Copyright Act 2000 — AI-specific provision absent।
    • 
        

        
Audit pipeline:

        
    
          
  • Sample ১০K generated outputs।
    • 
          
  • Compute CLIP/SBERT embedding।
    • 
          
  • Retrieve nearest training image/text।
    • 
          
  • Visual / text comparison।
    • 
          
  • Threshold report — "X% within 0.1 distance"।
    • 
        

        
Tools: memit, extract, FAISS, training-data-attribution libraries।

        
মূল উপলব্ধি: Memorization rare কিন্তু non-zero। Legal, ethical, technical mitigation parallel চাই। Generative AI mature হওয়ার এটি অপরিহার্য part।

      

    

  


  

    
 অনুশীলন

    
    
      
  1. 
        হিসাব: Real ও Generated দু'টিই scalar (১-D)। $\mu_r=0, \sigma_r^2=1$; $\mu_g=2, \sigma_g^2=1$। FID-র ১D analog কত?
        
    
          
          
    
            
    $(2-0)^2 + (1+1-2\sqrt{1 \cdot 1}) = 4 + 0 = 4$।
    
            
    Mean shift dominate কারণ variance same। Mean একই হলে variance mismatch দেখাত।
    
          
    
        
    
      
    2. 
      
  2. 
        Hands-on: Stable Diffusion-এ ১০০ ছবি generate করে FID-DINOv2 + CLIP score compute করুন (cleanfid library)।
        
    
          
          
    
            
    from cleanfid import fid
score = fid.compute_fid("./real", "./generated",
                        mode="clean", model_name="dinov2")
print("FD-DINOv2:", score)
    
          
    
        
    
      
    3. 
      
  3. 
        ভাবুন: বাংলা TTS-এর জন্য একটি evaluation rubric design করুন — naturalness, intelligibility, dialect accuracy, prosody।
        
    
          
          
    
            
      
              
    • MOS (Mean Opinion Score): 1-5 naturalness।
      • 
              
    • WER (Word Error Rate): back-transcribe।
      • 
              
    • SECS: speaker similarity (cosine of speaker embedding)।
      • 
              
    • Intelligibility: ৮-১০ Bangla word listener identify।
      • 
              
    • Pronunciation accuracy: proper noun (যশোর, ফরিদপুর)।
      • 
              
    • Prosody: question vs statement intonation।
      • 
            
    
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন

    
    

      Reproducible FID: pip install clean-fid — paper-quality FID যেকোনো dataset-এ।
      Colab।
    

  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ২৪ · Multimodal generation