পাঠ ৩১ · ৩৩-এর মধ্যে · মডিউল ৪
Home / AI Courses / MLOps / LLM cost

Cost optimization for LLM serving

LLM cost — token math to architecture
৭ মিনিট পড়া উচ্চ · Advanced Cost

এই পাঠে যা শিখবেন

  • Token cost math
  • Model routing + cascading
  • Caching strategies (exact + semantic)
  • Self-host crossover analysis

১ · Token cost basics

$$ \text{cost} = (n_{in} \cdot p_{in} + n_{out} \cdot p_{out}) \times \text{requests} $$

API rates 2025 (approx):

  • GPT-4o: $5/1M input, $15/1M output.
  • GPT-4o-mini: $0.15/1M input, $0.60/1M output.
  • Claude Sonnet: $3/$15.
  • Claude Haiku: $0.25/$1.25.
  • Self-hosted LLaMA 70B: ~$0.50/1M tokens.

২ · Cost calculation example

Python · Cost calculator 
def llm_cost(
    n_input_tokens: int,
    n_output_tokens: int,
    requests_per_day: int,
    model: str = "gpt-4o",
) -> dict:
    rates = {
        "gpt-4o": {"in": 5.00, "out": 15.00},  # per 1M tokens
        "gpt-4o-mini": {"in": 0.15, "out": 0.60},
        "claude-sonnet": {"in": 3.00, "out": 15.00},
        "claude-haiku": {"in": 0.25, "out": 1.25},
        "llama-70b-self": {"in": 0.50, "out": 0.50},
    }
    r = rates[model]
    cost_per_req = (
        (n_input_tokens * r["in"] / 1_000_000)
        + (n_output_tokens * r["out"] / 1_000_000)
    )
    daily = cost_per_req * requests_per_day
    monthly = daily * 30
    return {
        "model": model,
        "cost_per_request": round(cost_per_req, 5),
        "daily": round(daily, 2),
        "monthly": round(monthly, 2),
    }


# BD chatbot — Bangla token tax (3× English)
print(llm_cost(2000, 500, 10_000, "gpt-4o"))
# {'monthly': $4500}
print(llm_cost(2000, 500, 10_000, "gpt-4o-mini"))
# {'monthly': $135}

    


    

      30× cost difference GPT-4o vs mini — many simple tasks mini sufficient. Don't default GPT-4o.
    


    
৩ · Model routing (cascade)


    

      Easy → cheap model. Hard → escalate expensive.
    

    
    
      
  • Classifier "easy/hard" routes input.
    • 
      
  • Cheap model attempt; low confidence → escalate.
    • 
      
  • "Mixture of agents" — multiple cheap models combine.
    • 
    


    
৪ · Caching


    
Two cache types:

    
    
      
  • Exact match: request hash; common for FAQ-like.
    • 
      
  • Semantic: query embedding similar — reuse. 30-60% hit rate possible.
    • 
    


    

      

        
        Python · Semantic cache
      

      
import openai
import numpy as np
from typing import Optional

class SemanticCache:
    def __init__(self, threshold: float = 0.95):
        self.threshold = threshold
        self.cache = []  # list of (embedding, response)

    def _embed(self, text: str) -> np.ndarray:
        emb = openai.embeddings.create(
            input=text, model="text-embedding-3-small"
        ).data[0].embedding
        return np.array(emb)

    def get(self, query: str) -> Optional[str]:
        q_emb = self._embed(query)
        for emb, response in self.cache:
            sim = np.dot(q_emb, emb) / (np.linalg.norm(q_emb) * np.linalg.norm(emb))
            if sim > self.threshold:
                return response
        return None

    def set(self, query: str, response: str):
        emb = self._embed(query)
        self.cache.append((emb, response))


# Usage
cache = SemanticCache()

def cached_llm(query: str) -> str:
    cached = cache.get(query)
    if cached:
        return cached  # 0 cost
    # Otherwise, call LLM
    resp = openai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": query}],
    ).choices[0].message.content
    cache.set(query, resp)
    return resp

      

        
        
      

      

    


    

      Embedding-based similarity. Threshold tune carefully — too high (no hits), too low (wrong response). Production: Redis vector store.
    


    
৫ · Quantization


    
Self-host model: GPTQ/AWQ INT4 — 4× memory + speed.

    
(Lesson 20 details). Smaller GPU fits → cheaper instance.


    
৬ · Continuous batching (vLLM)


    
vLLM PagedAttention — variable-length output efficient. 5-10× throughput vs naive. GPU cost per request dramatically lower.


    
৭ · Self-host crossover


    

      $$ \text{breakeven\_RPS} = \frac{\text{API\_cost\_per\_token}}{\text{self\_host\_cost\_per\_token}} $$
    


    
BD context:

    
    
      
  • OpenAI mini: $0.50/1M tokens.
    • 
      
  • LLaMA 70B self-host: ~$0.50/1M (similar cost).
    • 
      
  • OpenAI GPT-4o: $10/1M.
    • 
      
  • Self-host crossover: ~50K daily requests.
    • 
    


    
৮ · KV cache


    
Multi-turn conversation — previous tokens KV state cache. Don't re-process. vLLM, TRT-LLM built-in.


    
৯ · Speculative decoding


    
Small "draft" model proposes; large model verifies. 2-3× speedup. vLLM/TGI supports.


    

      
        
        LLM cost optimization — layered approach

        
        Layer 1: Cache (semantic + exact) — eliminate calls 30-60%

        
        Layer 2: Model routing — easy → cheap, hard → escalate

        
        Layer 3: Prompt optimize — shorter context, fewer examples

        
        Layer 4: Self-host (vLLM, INT4 quantize) — at scale

        Combined effects: 5-10× cost reduction vs naive default.
      
      
LLM cost optimization layered — cache, route, optimize prompt, self-host. প্রতিটি 1.5-3× saving.

    


    

      Bangladesh chatbot reality — naive monthly $5K → optimized $500 possible. Cache + routing essential first steps.
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      প্র ০১"Self-host vs API — Bangladesh fintech-এর জন্য concrete numbers?"
      

        
Concrete decision-driving analysis.

        
Scenario:

        
    
          
  • 10K daily Bangla queries.
    • 
          
  • 2K input + 500 output tokens average.
    • 
          
  • Bangla token tax 3× → effective 6K input + 1.5K output English-equivalent.
    • 
        

        
OpenAI GPT-4o:

        
    
          
  • Daily: 10K × (6K × $5/M + 1.5K × $15/M) = $52.
    • 
          
  • Monthly: $1,560.
    • 
        

        
OpenAI GPT-4o-mini:

        
    
          
  • Daily: 10K × (6K × $0.15/M + 1.5K × $0.60/M) = $1.80.
    • 
          
  • Monthly: $54.
    • 
          
  • 30× cheaper, quality slightly less.
    • 
        

        
Self-hosted Llama 70B (vLLM, INT4):

        
    
          
  • 1× A100 instance: $1.5/hour × 24 × 30 = $1,080/month.
    • 
          
  • ~5K-10K req/day capacity.
    • 
          
  • Monthly: $1,080 fixed + ops cost.
    • 
          
  • Crossover with GPT-4o: yes, but mini cheaper.
    • 
        

        
Decision:

        
    
          
  • Volume small (10K/day): GPT-4o-mini, no self-host.
    • 
          
  • Volume large (100K+/day): self-host wins big.
    • 
          
  • Compliance forced: self-host regardless.
    • 
        

        
Hidden costs:

        
    
          
  • Self-host: ops engineer 0.2 FTE = $1K/month.
    • 
          
  • API: zero ops.
    • 
          
  • Real crossover often higher than naive token math suggest.
    • 
        

        
মূল উপলব্ধি: Most BD startup-এর জন্য — API-mini default. Self-host justify volume + compliance. Don't over-engineer initially.

      

    


    

      প্র ০২"Cache hit rate boost — practical tactics?"
      

        
Cache hit rate = saved cost; tactical optimization.

        
Boost tactics:

        
    
          
  • Normalize input: lowercase, strip whitespace, sort fields. Same intent → same key.
    • 
          
  • Semantic cache: embedding-based reuse similar queries.
    • 
          
  • FAQ extract: common questions pre-warm cache.
    • 
          
  • User tier: free-tier higher cache reuse; paid more fresh.
    • 
        

        
Threshold tuning:

        
    
          
  • 0.95+ — high precision, low recall.
    • 
          
  • 0.85 — more hits, occasional wrong answer.
    • 
          
  • Validate empirically.
    • 
        

        
Cache invalidation:

        
    
          
  • TTL — daily/weekly expire.
    • 
          
  • Source change — invalidate dependent cache.
    • 
        

        
Anti-pattern:

        
    
          
  • Over-aggressive cache — outdated information served.
    • 
          
  • Cache miss attribution wrong — semantic threshold issue often.
    • 
        

        
BD context:

        
    
          
  • Customer support FAQ — extreme cache hit (50-70%).
    • 
          
  • Personal recommendation — cache irrelevant.
    • 
        

        
মূল উপলব্ধি: Cache strategy use-case-specific. FAQ ideal; personal poor. Measure hit rate; tune threshold; pre-warm common.

      

    


    

      প্র ০৩"Outage fallback — API down হলে কী?"
      

        
OpenAI/Anthropic occasional outage; production resilience.

        
Strategies:

        
    
          
  • Multi-provider: primary OpenAI, fallback Anthropic. Switch on error.
    • 
          
  • Self-host fallback: degraded but available.
    • 
          
  • Rule-based fallback: simple keyword matching for common queries.
    • 
          
  • "Degrade gracefully": "I'm temporarily unavailable, please try later".
    • 
        

        
Implementation:

        
    
          
  • Circuit breaker per provider.
    • 
          
  • Retry with backoff.
    • 
          
  • Health check endpoint monitor.
    • 
        

        
Cost trade-off:

        
    
          
  • Multi-provider — vendor diversification, slightly more complex.
    • 
          
  • Self-host fallback — fixed cost regardless usage.
    • 
        

        
BD context — bKash chatbot:

        
    
          
  • Multi-provider primary; rule-based fallback bottom.
    • 
          
  • "Send via email" graceful escape.
    • 
        

        
Test:

        
    
          
  • Chaos engineering — block API, verify fallback.
    • 
          
  • Monthly drill.
    • 
        

        
মূল উপলব্ধি: LLM service depend external; resilience design. Multi-provider + degraded fallback minimum. Test failure paths.

      

    


    

      প্র ০৪"Cost attribution — multiple team-এর সাথে কীভাবে?"
      

        
Shared LLM infrastructure — cost allocation governance.

        
Approaches:

        
    
          
  • Per-request tagging: team/project label log; periodic reporting.
    • 
          
  • Separate API key: per-team OpenAI organization; clear billing.
    • 
          
  • Quota: monthly budget per team; lockout on overrun.
    • 
          
  • Showback: internal "billing" — chargeback culture.
    • 
        

        
Tools:

        
    
          
  • Helicone, Langfuse — usage tracking by user/project.
    • 
          
  • OpenAI organization-level usage API.
    • 
        

        
Governance:

        
    
          
  • Monthly cost dashboard share.
    • 
          
  • Top-cost prompts review.
    • 
          
  • Optimization SLA per team.
    • 
        

        
BD context:

        
    
          
  • Mid-size company — single API key + tagging usually.
    • 
          
  • Larger — separate org accounts.
    • 
        

        
Anti-pattern:

        
    
          
  • "Shared budget free for all" — runaway cost.
    • 
          
  • Without attribution — optimization no incentive.
    • 
        

        
মূল উপলব্ধি: Cost attribution = governance enabler. Without — accountability weak. Tagging cheap, billing organize-able. Day-1 setup essential.

      

    

  


  

    
 অনুশীলন

    
    
      
  1. Calculate: আপনার (real or imagined) LLM use case-এর monthly cost — different model.
        
    উপরের code adapt; vary RPS, tokens. 4-5 model compare table.
    
      
    2. 
      
  2. Cache: Semantic cache simple implementation — 100 query test, hit rate measure.
        
    Threshold 0.95, 0.90, 0.85 — accuracy vs hit rate trade-off.
    
      
    3. 
      
  3. চিন্তা: Bangla customer support optimization stack — 5 layers prioritize cost saving.
        
    
          
      
            
    • FAQ exact-match cache (40% hit).
      • 
            
    • Semantic cache (additional 20% hit).
      • 
            
    • GPT-4o-mini routing (vs 4o).
      • 
            
    • Prompt compression (Bangla input shorter).
      • 
            
    • Self-host LLaMA at high volume (5×+ scale).
      • 
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন

    
  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ৩০ · LLM eval & safety