পাঠ ২৪ · ২৮-এর মধ্যে · মডিউল ৪

Home / AI Courses / Generative AI / Multimodal

Multimodal — text + image + audio

Multimodal generation — CLIP, GPT-4V, Gemini, LLaVA

৭ মিনিট পড়া মধ্যবর্তী · Intermediate Vision-Language

এই পাঠে যা শিখবেন

Multimodal embedding — কীভাবে দু'টি modality একই space-এ
CLIP architecture ও contrastive loss
Multimodal LLM (GPT-4V, Gemini, LLaVA) কীভাবে image বুঝে
Joint generation pipeline ও practical use case

১ · Multimodal কেন গুরুত্বপূর্ণ

মানুষ unimodal নয়। চোখ, কান, ভাষা, স্পর্শ — সব মিলিয়ে concept গড়ে। AI-ও সেদিকে যাচ্ছে। Text-only LLM "এই ছবিতে কী আছে" বলতে পারে না। Image-only model "explain why this is funny" বলতে পারে না। Multimodal modelMultimodal modelএকাধিক input modality (text, image, audio, video) একসাথে process করার model। Joint reasoning সম্ভব। — দু'টোই করে।

২ · CLIP (Radford et al., OpenAI 2021) — foundation

CLIP-এর simple কিন্তু genius idea: web-এ image + alt-text বা caption pair সর্বত্র। ৪০০ million pair scrape করো, image encoder ও text encoder পাশাপাশি train করো — paired embedding-গুলো কাছে, unpaired দূরে।

Loss (InfoNCE): $$\mathcal{L} = -\log \frac{\exp(\text{sim}(\mathbf{i}, \mathbf{t}) / \tau)}{\sum_j \exp(\text{sim}(\mathbf{i}, \mathbf{t}_j) / \tau)}$$

$\mathbf{i}$ image embedding, $\mathbf{t}$ paired text। Batch-এর অন্য সব text negative।

কেন CLIP transformative

১) Zero-shot classification — train না করেই category list-এ image classify।
২) Image search — "natural language query" → ছবি।
৩) Stable Diffusion-এর text encoder = CLIP text।
৪) DALL·E 2-র prior — CLIP space-এ navigation।

৩ · Multimodal LLM — GPT-4V, Gemini, Claude 3, LLaVA

Architecture pattern:

Vision encoder (ViT, CLIP image) → image embedding।
Projection (linear / MLP / Q-Former) → LLM embedding space-এ map।
LLM প্রসঙ্গ-এ image token + text token mix → standard autoregressive generate।

Models:

GPT-4V (২০২৩): vision-augmented GPT-4, screenshot/chart বুঝে।
GPT-4o (২০২৪): native multimodal — text+image+audio একই model।
Gemini 1.5/2 (Google): ১M token context, video frames natively।
Claude 3.5 Sonnet: screenshot reasoning, computer use ability।
LLaVA (২০২৩, open-source): Llama + CLIP — community standard।
InternVL, Qwen-VL, MiniCPM-V: Chinese open-source contenders।
Llama 3.2 Vision: Meta-র multimodal Llama।

৪ · ImageBind (Meta 2023) — ৬ modality একসাথে

Meta-র ImageBind আরও দূর গেল — image-কে anchor করে text, audio, depth, thermal, IMU — সব একই space-এ। Image-text pair-এ train করেও audio-text alignment পাওয়া যায় (transitive)।

ImageBind = বহুভাষিক অভিধান। বাংলা-ইংরেজি ও ইংরেজি-আরবি জানলে — বাংলা-আরবি direct lookup ছাড়াও bridge করা যায়। Image-ই সেই "Esperanto"।

Multimodal embedding — সব modality contrastive loss-এ একই space-এ। CLIP, ImageBind, Gemini এই principle-এ।

৫ · CLIP দিয়ে hands-on — image search

Python · transformers (CLIP)

from transformers import CLIPProcessor, CLIPModel
from PIL import Image
import torch

model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

# Multilingual variant: "sentence-transformers/clip-ViT-B-32-multilingual-v1"
# (Bangla query support)

image = Image.open("dhaka_street.jpg")
candidates = ["a busy Dhaka street", "the Sundarbans tiger",
              "a Padma river boat", "a Bashundhara shopping mall"]

inputs = processor(text=candidates, images=image, return_tensors="pt", padding=True)
with torch.no_grad():
    outputs = model(**inputs)
probs = outputs.logits_per_image.softmax(dim=1)

for c, p in zip(candidates, probs[0]):
    print(f"{c}: {p:.3f}")

৬ · LLaVA / Qwen-VL দিয়ে image-to-text

Python · multimodal LLM

from transformers import AutoProcessor, LlavaForConditionalGeneration
import torch
from PIL import Image

model_id = "llava-hf/llava-1.5-7b-hf"
model = LlavaForConditionalGeneration.from_pretrained(
    model_id, torch_dtype=torch.float16, device_map="auto"
)
processor = AutoProcessor.from_pretrained(model_id)

image = Image.open("rickshaw.jpg")
prompt = "USER: \nবাংলায় এই ছবিটির বিস্তারিত বর্ণনা দিন।\nASSISTANT:"

inputs = processor(prompt, image, return_tensors="pt").to(model.device, torch.float16)
out = model.generate(**inputs, max_new_tokens=200, do_sample=True, temperature=0.5)
print(processor.decode(out[0], skip_special_tokens=True))

LLaVA-1.5 Bangla limited; Qwen2-VL ও InternVL Bangla অনেক ভাল। Production-এ Gemini Flash বা GPT-4o cheap+strong।

Multimodal LLM-এ jailbreak unique ঝুঁকি — image-এ embedded text via OCR system prompt override করতে পারে। "Prompt injection via image" growing concern।

ভাবনার প্রশ্ন

প্র ০১ CLIP-এর contrastive loss "InfoNCE" — কেন এটি এত কার্যকর? কী assumption-এ দাঁড়িয়ে? Failure mode কী?

InfoNCE (Oord 2018) representation learning-এ landmark loss। CLIP-এর success-এর ভিত্তি কিন্তু limitations গভীর।

InfoNCE-এর intuition:

Mutual information lower bound maximize।
Positive pair (paired image-text) closer; negative pair (random pairing) farther।
Batch-এ N-1 negative — implicit hard negative mining।
Temperature $\tau$ — distribution sharpness control।

কেন কাজ করে:

Self-supervised — manual label লাগে না।
Web data abundance (alt-text, caption) leverage।
Joint embedding space — multiple downstream task-এ portable।
Scale law — more data → better।

Assumptions:

Random batch sample → meaningful negative। Reality-এ "near-duplicate" pair অনেক — false negative।
One-to-one image-text correspondence। Reality-এ অনেক caption → অনেক image।
Text & image distribution similar। English-heavy → bias।

Failure modes:

Compositional reasoning: "red cube on blue sphere" — CLIP প্রায়ই blue cube on red sphere-ও closer score। Subject-object binding weak।
Text vs token: "Apple Inc" বনাম "apple fruit" — context-poor।
Counting: "3 dogs" বনাম "5 dogs" — CLIP ভুল।
Bangla / low-resource: training distribution-এ underrepresented; quality-drop ৩০-৫০%।
Bias: "doctor" → male image bias; "nurse" → female।

Modern improvements:

SigLIP (Google 2023): sigmoid loss, batch-size-independent। 80%+ ImageNet zero-shot।
DFN (Apple 2024): data filtering — quality > quantity।
EVA, OpenCLIP: open-source successors।
MetaCLIP, Recap: better caption curation।

Beyond contrastive:

Caption generation supervision (BLIP, BLIP-2)।
Masked image-text modeling (BEiT-3)।
Multimodal LLM joint training (LLaVA, Flamingo)।

মূল উপলব্ধি: InfoNCE simple কিন্তু powerful। Web-scale data + contrastive + transformer = foundation model। কিন্তু "alignment" surface-level — true compositional understanding এখনো গবেষণা।

প্র ০২ Multimodal LLM (GPT-4o, Gemini) "see" করে — কিন্তু কতটা সত্যিই দেখে? Hallucination, OCR error, spatial reasoning — practical limitations কী?

Multimodal LLM impressive demo দেখায় কিন্তু production-এ failure mode চিনতে হয়।

Hallucination types:

Object hallucination: ছবিতে নেই এমন বস্তু describe — "I see a dog" — কিন্তু dog নেই।
Attribute hallucination: red car-কে blue বলে।
Relational hallucination: "the cat is on the table" — অথচ cat নিচে।
OCR-overconfidence: blurry text-এ confident wrong reading।

Spatial reasoning weakness:

Counting: "how many people" — ৫+ এ accuracy পড়ে।
Position: "left of", "behind" — ৫০-৭০% accuracy।
Distance: "how far" — quantitative অনুমান weak।
Geometry: "is this triangle isosceles" — surprisingly bad।
POPE, MME, MMBench — diagnostic benchmark।

OCR limitations:

English clean text — ৯৫%+ accurate।
Handwriting — ৭০-৮৫%।
Bangla — ৫০-৭০% (varies provider; Gemini ভাল)।
Tabular data — structure preservation weak।
Math equation — partial।

Reasoning failures:

Cause-effect across panels (comic, infographic) — confused।
Chart reading — value approximate, not precise।
Map reading — direction often wrong।
Diagram (architecture, flowchart) — partial understanding।

Production safeguards:

OCR-specific tool (Tesseract, Google Vision) → MLLM hybrid।
Confidence calibration — "I'm not sure" output force।
Multiple sample → consistency check।
Specialized model (chart QA, doc QA, geometry solver) for niche।
Human-in-the-loop critical decisions।

Bangla-specific issue:

Bangla street sign, restaurant menu — OCR weak।
Cultural object recognition (পান্তা ভাত, মুড়ি ভর্তা) — Western-trained model বুঝে না।
Body language, dress (sari, panjabi) — context lacking।

Modern progress (২০২৪-২৫):

Gemini 2 — chart reading dramatic improvement।
Claude 3.5 Sonnet — screenshot-based UI navigation strong।
GPT-4o — multilingual OCR including Bangla improving।
Qwen2-VL, InternVL2 — Asian-language strong।

মূল উপলব্ধি: "MLLM sees" — metaphorical। Pattern-match strong, true visual reasoning weak। Augment, don't replace task-specific tools।

প্র ০৩ Bangladesh-এ একটি multimodal app — Bangla menu OCR + price extract + dietary info। কোন stack? Edge cases কী?

Restaurant menu app বাংলাদেশে বিশাল need — tourists, Bangladeshi expat, dietary restriction। Real production challenge।

Stack options:

Option A · Pure MLLM (Gemini 2.0 Flash, GPT-4o): photo upload → "extract menu items, price, allergens"। Simple, $0.001-0.01 per image।
Option B · Hybrid (OCR + LLM): Google Vision/Tesseract Bangla OCR → text → LLM structure ও enrich।
Option C · Fine-tuned (LLaVA-Bangla): own data, low cost long-term, infra heavy।

Recommendation: Option B (hybrid)

OCR specialized — Bangla text accuracy ৮৫-৯৫%।
LLM downstream — semantic enrichment।
Cost predictable, debug সহজ।

Pipeline:

User photo → image quality check (blur, lighting)।
Layout detection — text region ও price column identify।
Bangla OCR (Google Vision or self-hosted)।
LLM (Claude Haiku, Gemini Flash) — structured JSON extract।
Enrichment — dietary tags (halal, vegetarian), allergen (peanut, dairy)।
Translation — English/Hindi for tourist।
Knowledge graph — popular dish lookup, price comparison।

Edge cases:

Handwritten chalkboard menu: OCR weak। MLLM-এ direct আবার hallucinate।
Bilingual menu (Bangla + English): language detection per item।
Numerals (১২০ vs 120): Bangla digit conversion।
Dish name ambiguity: "ভর্তা" — what kind? Context-aware।
Photo angle: tilted, partial — perspective correction।
Lamination glare: common at street food spots।
Stained menu: tea/oil obscure text।
Multi-page menu: photo stitching।
Updated price: handwritten override on printed।

Knowledge curation:

Bangladeshi cuisine dictionary — ১০,০০০+ dish।
Halal/haram database (pork product detection)।
Spice level (mild/medium/hot)।
Common allergen mapping।
Price range geo-aware (Dhaka vs Sylhet vs rural)।

UX considerations:

Slow connection — progressive load।
Offline mode — cached recent menu।
Photo zoom on uncertain item।
User correction feedback loop।
Voice TTS for elderly user।

Privacy:

Photo on-device process if possible।
Personal dietary restriction — local storage।
Restaurant data crowd-sourced — moderation।

Monetization:

Freemium — basic free, advanced (allergen, calorie) premium।
Restaurant partner — verified menu listing।
Ad to local food delivery (foodpanda, Pathao Food)।

মূল উপলব্ধি: Multimodal AI infrastructure ready — কিন্তু product success Bangladeshi context-এ data + UX-এ। Edge case-এ যিনি ভাল — তিনি win।

প্র ০৪ Bangladesh-এর ৫০টি ভাষাভাষী dialect — Sylheti, Chittagonian, Rohingya — multimodal AI কীভাবে handle? Equity, low-resource challenge।

Bangladesh "Bangla" monolingual নয়। Sylheti ১৫M speaker, Chittagonian ১৩M, Rohingya ১M+ refugee, ৪৫+ minority language। AI-এ অধিকাংশ invisible।

Why this matters:

UN Sustainable Development — language equity।
Healthcare — patient mother-tongue critical।
Disaster response — flood warning local language।
Education — early childhood mother-tongue learning।
Justice — defendant language right।

Current state:

Bangla: Bangladesh standard — model decent।
Sylheti: separate ISO 639-3 code (syl); written Sylhet Nagari ও Bangla script। AI virtually nonexistent।
Chittagonian: ISO 639-3 (ctg); spoken-only। Almost no NLP।
Rohingya: ISO 639-3 (rhg); UNHCR script standard। Minimal AI।
Garo, Chakma, Marma: own script; almost zero।

Multimodal angle:

Speech-to-speech translation — UNHCR refugee context urgent।
Image OCR — Sylhet Nagari, Chakma script preservation।
Sign language (BdSL — Bangladeshi Sign Language) — visual-language model।
Cultural object recognition — tribal artifact, dress, festival।

Technical challenges:

Data scarcity: Common Crawl-এ <0.001%। Manual collection।
Code-switching: Sylheti speaker often switch Bangla mid-sentence।
Script multiplicity: same language multiple script।
Phonology: Sylheti tone unlike Bangla।
Morphology: agglutinative complexity।
Standardization: spelling not standardized।

Approaches:

Cross-lingual transfer: Bangla → Sylheti via shared Indo-Aryan root।
Massively multilingual model (NLLB-200, Aya): low-resource embed।
Few-shot learning: ১০০ example দিয়ে adaptation।
Community-driven data collection: participatory ML (Common Voice, Lacuna Fund)।
Self-supervised speech: wav2vec, Whisper — unlabeled audio।
Visual grounding: image-language pair where text scarce — image bridge।

Initiatives to follow:

BUET CSE: Bangla NLP research।
SIL International: minority language documentation।
UNHCR + Translators Without Borders: Rohingya MT।
Gates Foundation, Lacuna Fund: low-resource grant।
Mozilla Common Voice: volunteer audio।
Masakhane (African): model for Bangladesh community।

Ethical principles:

Community consent ও control।
Sacred / private content protection।
Benefit return to community।
Scripts, oral tradition respect।

Multimodal advantage: Visual + audio — script না জানলেও speech + image input দিয়ে interaction। Inclusive design।

মূল উপলব্ধি: "AI for Bangladesh" — শুধু standard Bangla নয়। সব ভাষাভাষীর jaiga — দূর্বল ভাষা hold করতে multimodal best tool। Tech equity = social equity।

অনুশীলন

হাতে-কলমে: CLIP দিয়ে আপনার ফোনের ১০টি ছবি-তে "outdoor", "people", "food", "vehicle" — কোনটি best match?

candidates = ["outdoor scene", "people", "food", "vehicle"]
for img_path in glob.glob("photos/*.jpg"):
    img = Image.open(img_path)
    inputs = processor(text=candidates, images=img, return_tensors="pt", padding=True)
    probs = model(**inputs).logits_per_image.softmax(dim=1)[0]
    print(img_path, "→", candidates[probs.argmax()])

হিসাব: CLIP ৪০০M pair, batch 32K-এ train। প্রতি step-এ কত contrastive comparison?

Batch-এ ৩২,৭৬৮ pair × ৩২,৭৬৭ negative = ~১.১ billion comparison per step। ~৩২ epoch × ~১২,০০০ step ≈ ৪২৫ trillion comparison। Compute = ১২৫৬ TPU days।
ভাবুন: Bangla audiobook narrator-এর জন্য একটি multimodal feature design করুন — image (book cover) + voice (narrator sample) → narration style।
- Cover image → CLIP embedding → mood (mystery, romance)।
- Narrator sample → speaker embedding (XTTS)।
- Joint conditioning — TTS-এ mood+voice combine।
- "Sad mood" → slower pace, lower pitch।