পাঠ ২৮ · ৩৩-এর মধ্যে · মডিউল ৪
Home / AI Courses / MLOps / LLMOps essentials

LLMOps — কী আলাদা

LLMOps — what's different from classical MLOps
৮ মিনিট পড়া উচ্চ · Advanced Concept

এই পাঠে যা শিখবেন

  • LLMOps vs classical MLOps — key differences
  • Foundation model vs fine-tune vs RAG decision
  • Vector DB landscape
  • Bangla LLM-specific challenges

১ · কেন আলাদা

Classical ML: train custom model, deploy, monitor. LLM-এ অনেক assumption ভাঙে।

২ · ৭টি core difference

LLMOps differences
  1. Non-determinism: temperature, top-p — same prompt different output.
  2. Prompt as artifact: code-like, version controlled, tested.
  3. Foundation model dependency: rarely train from scratch; fine-tune or use API.
  4. RAG infrastructure: vector DB, retrieval, context management.
  5. Token cost: input + output tokens, model-tier pricing.
  6. Hallucinations: confidently wrong; classical metric inadequate.
  7. Streaming: token-by-token return; latency definition shifted.

৩ · Build vs buy

  • API (OpenAI, Anthropic, Cohere):
    • Pros: zero infra, latest model, fast.
    • Cons: per-token cost, data privacy, vendor lock.
  • Self-host open model (LLaMA, Mistral, Qwen):
    • Pros: data control, customizable, eventually cheaper at scale.
    • Cons: GPU cost, ops complexity, behind frontier.
  • Fine-tune:
    • Pros: domain-specific quality, smaller model possible.
    • Cons: training cost, drift management.

৪ · RAG (Retrieval Augmented Generation)

LLM-এর knowledge cut-off + hallucinations issue. RAG = retrieve relevant context → LLM answer.

  • Document ingestion → chunk → embed → vector DB.
  • Query: embed query → vector search → top-k chunks.
  • Prompt: "Given context: [...], answer: ..."।

৫ · Vector DB landscape

  • FAISS (Meta): in-memory library; fast; no persistence built-in.
  • pgvector (Postgres): embedded SQL extension; familiar; scalable.
  • Qdrant: Rust-built, fast, REST API.
  • Weaviate: GraphQL, schema-aware, hybrid search.
  • Pinecone: SaaS, easiest, pricey.
  • Chroma: simple, dev-friendly.
  • Milvus: Cloud-native, K8s-friendly, large scale.

৬ · Simple RAG example

Python · RAG with pgvector 
import openai
import psycopg2
from psycopg2.extras import execute_values

# 1. Ingest documents
def ingest(docs: list[str], conn):
    for doc in docs:
        emb = openai.embeddings.create(
            input=doc, model="text-embedding-3-small"
        ).data[0].embedding
        conn.execute(
            "INSERT INTO docs (text, embedding) VALUES (%s, %s)",
            (doc, emb),
        )

# 2. Retrieve top-k
def retrieve(query: str, k: int = 5, conn) -> list[str]:
    emb = openai.embeddings.create(
        input=query, model="text-embedding-3-small"
    ).data[0].embedding
    rows = conn.execute(
        """SELECT text FROM docs
           ORDER BY embedding <=> %s::vector
           LIMIT %s""",
        (emb, k),
    ).fetchall()
    return [r[0] for r in rows]

# 3. Generate
def answer(query: str, conn) -> str:
    chunks = retrieve(query, conn=conn)
    context = "\n".join(chunks)
    prompt = f"""Context:
{context}

Question: {query}
Answer:"""
    resp = openai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
    )
    return resp.choices[0].message.content

    


    

      Minimal RAG: embed → store → retrieve → generate. Production-এ chunking strategy, hybrid search, reranker — সবই add.
    


    
৭ · Hallucinations


    

      LLM confidently wrong answer. Classical metric adequately capture না।
    

    
    
      
  • Detection: factuality check (knowledge graph), self-consistency, retrieval grounding.
    • 
      
  • Mitigation: RAG, low temperature, instruct model "say I don't know".
    • 
      
  • Eval: human review, LLM-as-judge (Lesson 30).
    • 
    


    
৮ · Bangla LLM specifics


    
    
      
  • Tokenization — Bangla character-rich; many model token-inefficient (300% more tokens).
    • 
      
  • Models: BanglaT5, BanglaGPT, BanglaBERT, mBART, BanglaLLaMA.
    • 
      
  • Eval scarce — Bangla benchmarks limited.
    • 
      
  • Cultural context — Western model often miss.
    • 
    


    

      
        
        LLMOps stack — RAG architecture

        
        User query

        
        Embed query
        embedding model

        
        Vector DB
        retrieve top-k

        
        Reranker
        optional

        
        Prompt template
        versioned

        
        LLM (API/local)
        streaming

        
        Guardrails
        safety, citations

        
        User

        
        
        
        
        
        
        

        Each component own monitoring + cost + version. Classical ML stack-এর সাথে quite different.
      
      
Production RAG architecture — embed/retrieve/rerank/prompt/LLM/guardrails — many moving parts each independently versioned.

    


    

      Bangladesh-এ LLMOps adoption fast — chatbot, customer support, knowledge base. Bangla LLM eval ও cost-control critical.
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      প্র ০১"Foundation model vs fine-tune vs RAG — কখন কোনটা?"
      

        
Choice depends task + data + budget.

        
Foundation model (out-of-box):

        
    
          
  • Pros: zero data needed, fast iterate.
    • 
          
  • Cons: generic; specialized task moderate quality.
    • 
          
  • Use: prototyping, simple tasks (summarize, translate).
    • 
        

        
RAG:

        
    
          
  • Pros: knowledge grounding, no training, easy update knowledge.
    • 
          
  • Cons: retrieval quality dependent, latency added.
    • 
          
  • Use: question-answering with proprietary documents.
    • 
        

        
Fine-tune:

        
    
          
  • Pros: domain-specific quality, faster inference (smaller model).
    • 
          
  • Cons: training data needed, retrain cost.
    • 
          
  • Use: classification, structured output, style adaptation.
    • 
        

        
Decision tree:

        
    
          
  • "Knowledge facts (changing)" → RAG.
    • 
          
  • "Style/format adaptation" → fine-tune.
    • 
          
  • "Generic task" → foundation.
    • 
          
  • "Complex domain" → fine-tune + RAG.
    • 
        

        
BD example — Bangla customer support:

        
    
          
  • Knowledge: company FAQ → RAG.
    • 
          
  • Tone: friendly Bangla → fine-tune.
    • 
          
  • Both combined: best result.
    • 
        

        
মূল উপলব্ধি: RAG-first usually. Fine-tune for style/format. Foundation for prototype. Combined production stack-এ different layers.

      

    


    

      প্র ০২"In-house LLM hosting vs API — Bangladesh fintech-এ?"
      

        
Strategic decision — privacy, cost, latency.

        
API (OpenAI, Anthropic):

        
    
          
  • Pros: best quality, no infra, fast iterate.
    • 
          
  • Cons: data privacy (financial), per-token cost, latency to US.
    • 
        

        
Self-host:

        
    
          
  • Pros: data control, fixed cost at scale, on-prem possible.
    • 
          
  • Cons: GPU expensive, ops, lower model quality.
    • 
        

        
Hybrid:

        
    
          
  • Sensitive — self-host (anonymized data API).
    • 
          
  • Non-sensitive — API.
    • 
        

        
Bangladesh Bank ICT:

        
    
          
  • Customer data abroad — restrictive.
    • 
          
  • Self-host or anonymize before API.
    • 
        

        
Cost crossover:

        
    
          
  • OpenAI GPT-4o: $5/1M input + $15/1M output tokens.
    • 
          
  • 1M conversations/day × 1K tokens = 1B tokens/day = $5K-15K/day.
    • 
          
  • Self-host: ~$5K/month cluster fixed.
    • 
          
  • Crossover at ~50K conversations/day.
    • 
        

        
BD reality:

        
    
          
  • Most BD startups API (cost manageable at low volume).
    • 
          
  • Banks/fintech moving self-host for compliance + scale.
    • 
          
  • Hybrid common compromise.
    • 
        

        
মূল উপলব্ধি: Volume + privacy → self-host; agility + low-volume → API. Hybrid practical for most production.

      

    


    

      প্র ০৩"Bangla LLM tokenization inefficiency কতটা painful?"
      

        
Bangla character-rich; many models trained primarily on English.

        
Tokenization tax:

        
    
          
  • English: ~0.75 tokens/word.
    • 
          
  • Bangla on English-tuned tokenizer: ~3-5 tokens/word.
    • 
          
  • Translation: 1 Bangla word = 4× cost vs English.
    • 
        

        
Models with better Bangla:

        
    
          
  • GPT-4o — improved multilingual; still 2-3× English.
    • 
          
  • Claude 3 — comparable.
    • 
          
  • Gemini — Google's Bangla focus.
    • 
          
  • BanglaGPT, BanglaT5 — Bangla-native tokenizer; 1× rate.
    • 
        

        
Cost impact:

        
    
          
  • $1000/month English → $3000-5000/month Bangla.
    • 
          
  • Significant for scale.
    • 
        

        
Mitigation:

        
    
          
  • Translate to English (lossy).
    • 
          
  • Bangla-native model (lower quality).
    • 
          
  • Custom tokenizer + fine-tune.
    • 
          
  • Hybrid: structured fields English; user input Bangla.
    • 
        

        
BD platform reality:

        
    
          
  • Customer support chatbot — Bangla-heavy, cost surprise often.
    • 
          
  • Budget projection English-rate → 3× actual.
    • 
        

        
মূল উপলব্ধি: Bangla token cost asymmetry critical for BD ops planning. Realistic estimate at start. Bangla-native model evaluate seriously.

      

    


    

      প্র ০৪"Hallucination — production-grade mitigation strategy?"
      

        
Hallucination single biggest LLM production risk.

        
Sources:

        
    
          
  • Knowledge gap — model doesn't know, fabricates.
    • 
          
  • Reasoning failure — confident wrong logic.
    • 
          
  • Prompt ambiguity — model assumes.
    • 
        

        
Mitigation layers:

        
(১) Retrieval grounding:

        
    
          
  • "Answer from context only".
    • 
          
  • Citations forced — answer cite source.
    • 
          
  • Mostly RAG-based.
    • 
        

        
(২) Low temperature:

        
    
          
  • Temperature 0.0-0.2 for fact-based.
    • 
          
  • Reduces creativity, increases consistency.
    • 
        

        
(৩) Self-verification:

        
    
          
  • "Check your answer for accuracy" follow-up prompt.
    • 
          
  • "I don't know" allowed in instruction.
    • 
        

        
(৪) Output guardrails:

        
    
          
  • NeMo Guardrails, LlamaGuard.
    • 
          
  • Citation verification — generated facts vs source.
    • 
        

        
(৫) Confidence scores:

        
    
          
  • Multi-sample agreement = confidence.
    • 
          
  • Low confidence → escalate human.
    • 
        

        
(৬) Human-in-loop:

        
    
          
  • Critical decision — review queue.
    • 
          
  • Spot-check sample.
    • 
        

        
BD context — financial advice:

        
    
          
  • Hallucinated rates / regulations dangerous.
    • 
          
  • Strict RAG grounding.
    • 
          
  • Disclaimers.
    • 
          
  • Low-confidence → human.
    • 
        

        
মূল উপলব্ধি: Hallucination = layered defense. RAG + temperature + guardrails + human review. Single technique insufficient. Risk-tier task accordingly.

      

    

  


  

    
 অনুশীলন

    
    
      
  1. Mini RAG: Local pgvector + OpenAI embedding + GPT-4o-mini — 10 docs ingest, query.
        
    Postgres pgvector enable. Sample text ingest. Query similar retrieve.
    
      
    2. 
      
  2. Compare: Same prompt 5 times — output variation observe.
        
    Temperature 0.7 visible variation; 0.0 stable. Determinism trade-off.
    
      
    3. 
      
  3. চিন্তা: Bangla customer support chatbot architecture design — vendor + RAG + guardrails.
        
    
          
      
            
    • OpenAI GPT-4o + custom Bangla prompt template.
      • 
            
    • RAG: Bangla FAQ pgvector.
      • 
            
    • Guardrails: PII filter, max length, profanity check.
      • 
            
    • Confidence scoring → human escalation.
      • 
            
    • Logging: full conversation analytics.
      • 
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন

    
  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ২৭ · Prometheus & Grafana