পাঠ ৩১ · ৩৩-এর মধ্যে · মডিউল ৪
Home / AI Courses / MLOps / LLM cost

Cost optimization for LLM serving

LLM cost — token math to architecture
৭ মিনিট পড়া উচ্চ · Advanced Cost

এই পাঠে যা শিখবেন

  • Token cost math
  • Model routing + cascading
  • Caching strategies (exact + semantic)
  • Self-host crossover analysis

১ · Token cost basics

$$ \text{cost} = (n_{in} \cdot p_{in} + n_{out} \cdot p_{out}) \times \text{requests} $$

API rates 2025 (approx):

  • GPT-4o: $5/1M input, $15/1M output।
  • GPT-4o-mini: $0.15/1M input, $0.60/1M output।
  • Claude Sonnet: $3/$15।
  • Claude Haiku: $0.25/$1.25।
  • Self-hosted LLaMA 70B: ~$0.50/1M tokens।

২ · Cost calculation example

Python · Cost calculator 
def llm_cost(
    n_input_tokens: int,
    n_output_tokens: int,
    requests_per_day: int,
    model: str = "gpt-4o",
) -> dict:
    rates = {
        "gpt-4o": {"in": 5.00, "out": 15.00},  # per 1M tokens
        "gpt-4o-mini": {"in": 0.15, "out": 0.60},
        "claude-sonnet": {"in": 3.00, "out": 15.00},
        "claude-haiku": {"in": 0.25, "out": 1.25},
        "llama-70b-self": {"in": 0.50, "out": 0.50},
    }
    r = rates[model]
    cost_per_req = (
        (n_input_tokens * r["in"] / 1_000_000)
        + (n_output_tokens * r["out"] / 1_000_000)
    )
    daily = cost_per_req * requests_per_day
    monthly = daily * 30
    return {
        "model": model,
        "cost_per_request": round(cost_per_req, 5),
        "daily": round(daily, 2),
        "monthly": round(monthly, 2),
    }


# BD chatbot — Bangla token tax (3× English)
print(llm_cost(2000, 500, 10_000, "gpt-4o"))
# {'monthly': $4500}
print(llm_cost(2000, 500, 10_000, "gpt-4o-mini"))
# {'monthly': $135}

    


    

      30× cost difference GPT-4o vs mini — many simple tasks mini sufficient। Don't default GPT-4o।
    


    
৩ · Model routing (cascade)


    

      Easy → cheap model। Hard → escalate expensive।
    

    
    
      
  • Classifier "easy/hard" routes input।
    • 
      
  • Cheap model attempt; low confidence → escalate।
    • 
      
  • "Mixture of agents" — multiple cheap models combine।
    • 
    


    
৪ · Caching


    
Two cache types:

    
    
      
  • Exact match: request hash; common for FAQ-like।
    • 
      
  • Semantic: query embedding similar — reuse। 30-60% hit rate possible।
    • 
    


    

      

        
        Python · Semantic cache
      

      
import openai
import numpy as np
from typing import Optional

class SemanticCache:
    def __init__(self, threshold: float = 0.95):
        self.threshold = threshold
        self.cache = []  # list of (embedding, response)

    def _embed(self, text: str) -> np.ndarray:
        emb = openai.embeddings.create(
            input=text, model="text-embedding-3-small"
        ).data[0].embedding
        return np.array(emb)

    def get(self, query: str) -> Optional[str]:
        q_emb = self._embed(query)
        for emb, response in self.cache:
            sim = np.dot(q_emb, emb) / (np.linalg.norm(q_emb) * np.linalg.norm(emb))
            if sim > self.threshold:
                return response
        return None

    def set(self, query: str, response: str):
        emb = self._embed(query)
        self.cache.append((emb, response))


# Usage
cache = SemanticCache()

def cached_llm(query: str) -> str:
    cached = cache.get(query)
    if cached:
        return cached  # 0 cost
    # Otherwise, call LLM
    resp = openai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": query}],
    ).choices[0].message.content
    cache.set(query, resp)
    return resp

      

        
        
      

      

    


    

      Embedding-based similarity। Threshold tune carefully — too high (no hits), too low (wrong response)। Production: Redis vector store।
    


    
৫ · Quantization


    
Self-host model: GPTQ/AWQ INT4 — 4× memory + speed।

    
(Lesson 20 details)। Smaller GPU fits → cheaper instance।


    
৬ · Continuous batching (vLLM)


    
vLLM PagedAttention — variable-length output efficient। 5-10× throughput vs naive। GPU cost per request dramatically lower।


    
৭ · Self-host crossover


    

      $$ \text{breakeven\_RPS} = \frac{\text{API\_cost\_per\_token}}{\text{self\_host\_cost\_per\_token}} $$
    


    
BD context:

    
    
      
  • OpenAI mini: $0.50/1M tokens।
    • 
      
  • LLaMA 70B self-host: ~$0.50/1M (similar cost)।
    • 
      
  • OpenAI GPT-4o: $10/1M।
    • 
      
  • Self-host crossover: ~50K daily requests।
    • 
    


    
৮ · KV cache


    
Multi-turn conversation — previous tokens KV state cache। Don't re-process। vLLM, TRT-LLM built-in।


    
৯ · Speculative decoding


    
Small "draft" model proposes; large model verifies। 2-3× speedup। vLLM/TGI supports।


    

      
        
        LLM cost optimization — layered approach

        
        Layer 1: Cache (semantic + exact) — eliminate calls 30-60%

        
        Layer 2: Model routing — easy → cheap, hard → escalate

        
        Layer 3: Prompt optimize — shorter context, fewer examples

        
        Layer 4: Self-host (vLLM, INT4 quantize) — at scale

        Combined effects: 5-10× cost reduction vs naive default।
      
      
LLM cost optimization layered — cache, route, optimize prompt, self-host। প্রতিটি 1.5-3× saving।

    


    

      Bangladesh chatbot reality — naive monthly $5K → optimized $500 possible। Cache + routing essential first steps।
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      প্র ০১"Self-host vs API — Bangladesh fintech-এর জন্য concrete numbers?"
      

        
Concrete decision-driving analysis।

        
Scenario:

        
    
          
  • 10K daily Bangla queries।
    • 
          
  • 2K input + 500 output tokens average।
    • 
          
  • Bangla token tax 3× → effective 6K input + 1.5K output English-equivalent।
    • 
        

        
OpenAI GPT-4o:

        
    
          
  • Daily: 10K × (6K × $5/M + 1.5K × $15/M) = $52।
    • 
          
  • Monthly: $1,560।
    • 
        

        
OpenAI GPT-4o-mini:

        
    
          
  • Daily: 10K × (6K × $0.15/M + 1.5K × $0.60/M) = $1.80।
    • 
          
  • Monthly: $54।
    • 
          
  • 30× cheaper, quality slightly less।
    • 
        

        
Self-hosted Llama 70B (vLLM, INT4):

        
    
          
  • 1× A100 instance: $1.5/hour × 24 × 30 = $1,080/month।
    • 
          
  • ~5K-10K req/day capacity।
    • 
          
  • Monthly: $1,080 fixed + ops cost।
    • 
          
  • Crossover with GPT-4o: yes, but mini cheaper।
    • 
        

        
Decision:

        
    
          
  • Volume small (10K/day): GPT-4o-mini, no self-host।
    • 
          
  • Volume large (100K+/day): self-host wins big।
    • 
          
  • Compliance forced: self-host regardless।
    • 
        

        
Hidden costs:

        
    
          
  • Self-host: ops engineer 0.2 FTE = $1K/month।
    • 
          
  • API: zero ops।
    • 
          
  • Real crossover often higher than naive token math suggest।
    • 
        

        
মূল উপলব্ধি: Most BD startup-এর জন্য — API-mini default। Self-host justify volume + compliance। Don't over-engineer initially।

      

    


    

      প্র ০২"Cache hit rate boost — practical tactics?"
      

        
Cache hit rate = saved cost; tactical optimization।

        
Boost tactics:

        
    
          
  • Normalize input: lowercase, strip whitespace, sort fields। Same intent → same key।
    • 
          
  • Semantic cache: embedding-based reuse similar queries।
    • 
          
  • FAQ extract: common questions pre-warm cache।
    • 
          
  • User tier: free-tier higher cache reuse; paid more fresh।
    • 
        

        
Threshold tuning:

        
    
          
  • 0.95+ — high precision, low recall।
    • 
          
  • 0.85 — more hits, occasional wrong answer।
    • 
          
  • Validate empirically।
    • 
        

        
Cache invalidation:

        
    
          
  • TTL — daily/weekly expire।
    • 
          
  • Source change — invalidate dependent cache।
    • 
        

        
Anti-pattern:

        
    
          
  • Over-aggressive cache — outdated information served।
    • 
          
  • Cache miss attribution wrong — semantic threshold issue often।
    • 
        

        
BD context:

        
    
          
  • Customer support FAQ — extreme cache hit (50-70%)।
    • 
          
  • Personal recommendation — cache irrelevant।
    • 
        

        
মূল উপলব্ধি: Cache strategy use-case-specific। FAQ ideal; personal poor। Measure hit rate; tune threshold; pre-warm common।

      

    


    

      প্র ০৩"Outage fallback — API down হলে কী?"
      

        
OpenAI/Anthropic occasional outage; production resilience।

        
Strategies:

        
    
          
  • Multi-provider: primary OpenAI, fallback Anthropic। Switch on error।
    • 
          
  • Self-host fallback: degraded but available।
    • 
          
  • Rule-based fallback: simple keyword matching for common queries।
    • 
          
  • "Degrade gracefully": "I'm temporarily unavailable, please try later"।
    • 
        

        
Implementation:

        
    
          
  • Circuit breaker per provider।
    • 
          
  • Retry with backoff।
    • 
          
  • Health check endpoint monitor।
    • 
        

        
Cost trade-off:

        
    
          
  • Multi-provider — vendor diversification, slightly more complex।
    • 
          
  • Self-host fallback — fixed cost regardless usage।
    • 
        

        
BD context — bKash chatbot:

        
    
          
  • Multi-provider primary; rule-based fallback bottom।
    • 
          
  • "Send via email" graceful escape।
    • 
        

        
Test:

        
    
          
  • Chaos engineering — block API, verify fallback।
    • 
          
  • Monthly drill।
    • 
        

        
মূল উপলব্ধি: LLM service depend external; resilience design। Multi-provider + degraded fallback minimum। Test failure paths।

      

    


    

      প্র ০৪"Cost attribution — multiple team-এর সাথে কীভাবে?"
      

        
Shared LLM infrastructure — cost allocation governance।

        
Approaches:

        
    
          
  • Per-request tagging: team/project label log; periodic reporting।
    • 
          
  • Separate API key: per-team OpenAI organization; clear billing।
    • 
          
  • Quota: monthly budget per team; lockout on overrun।
    • 
          
  • Showback: internal "billing" — chargeback culture।
    • 
        

        
Tools:

        
    
          
  • Helicone, Langfuse — usage tracking by user/project।
    • 
          
  • OpenAI organization-level usage API।
    • 
        

        
Governance:

        
    
          
  • Monthly cost dashboard share।
    • 
          
  • Top-cost prompts review।
    • 
          
  • Optimization SLA per team।
    • 
        

        
BD context:

        
    
          
  • Mid-size company — single API key + tagging usually।
    • 
          
  • Larger — separate org accounts।
    • 
        

        
Anti-pattern:

        
    
          
  • "Shared budget free for all" — runaway cost।
    • 
          
  • Without attribution — optimization no incentive।
    • 
        

        
মূল উপলব্ধি: Cost attribution = governance enabler। Without — accountability weak। Tagging cheap, billing organize-able। Day-1 setup essential।

      

    

  


  

    
 অনুশীলন

    
    
      
  1. Calculate: আপনার (real or imagined) LLM use case-এর monthly cost — different model।
        
    উপরের code adapt; vary RPS, tokens। 4-5 model compare table।
    
      
    2. 
      
  2. Cache: Semantic cache simple implementation — 100 query test, hit rate measure।
        
    Threshold 0.95, 0.90, 0.85 — accuracy vs hit rate trade-off।
    
      
    3. 
      
  3. চিন্তা: Bangla customer support optimization stack — 5 layers prioritize cost saving।
        
    
          
      
            
    • FAQ exact-match cache (40% hit)।
      • 
            
    • Semantic cache (additional 20% hit)।
      • 
            
    • GPT-4o-mini routing (vs 4o)।
      • 
            
    • Prompt compression (Bangla input shorter)।
      • 
            
    • Self-host LLaMA at high volume (5×+ scale)।
      • 
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন

    
  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ৩০ · LLM eval & safety