পাঠ ২৯ · ৩৩-এর মধ্যে · মডিউল ৪

Home / AI Courses / MLOps / Prompt management

Prompt management ও versioning

Prompts as code — version, test, deploy

৭ মিনিট পড়া উচ্চ · Advanced Concept

এই পাঠে যা শিখবেন

Prompt-এর storage strategies
Templating tools
Versioning + A/B testing
Regression suite

১ · "Prompt = code" mindset

Prompt code-এর মতো production artifact। Hardcoded string-এ বসিয়ে দিলে — change track নেই, A/B test নেই, regression catch হবে না।

২ · Storage approaches

Git-tracked files: simplest; .txt or .yaml in repo। Pros: Git workflow। Cons: deploy required for change।
Database: dynamic update; versioned। Pros: hot-reload। Cons: schema, governance।
Prompt registry SaaS: Langfuse, PromptLayer, Helicone — purpose-built। Pros: UI, A/B built-in। Cons: cost।

৩ · Templating

Python · Prompt template

from jinja2 import Template
from dataclasses import dataclass
import yaml

@dataclass
class PromptVersion:
    name: str
    version: str
    template: str
    description: str

# Load from YAML — git-tracked
PROMPTS = yaml.safe_load(open("prompts.yaml"))

bangla_summarize = PromptVersion(**PROMPTS["bangla_summarize_v3"])

def render(prompt: PromptVersion, **vars) -> str:
    return Template(prompt.template).render(**vars)

# Usage
prompt_text = render(
    bangla_summarize,
    article="Bangladesh-এর IT সেক্টর এখন...",
    max_words=100,
)

# Track in logging
import openai
resp = openai.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": prompt_text}],
)

logger.info({
    "prompt_name": bangla_summarize.name,
    "prompt_version": bangla_summarize.version,
    "input_tokens": resp.usage.prompt_tokens,
    "output_tokens": resp.usage.completion_tokens,
})

Pattern: prompts YAML, version field, log version + tokens। Production tracing connect specific prompt-version-এ।

৪ · prompts.yaml example

YAML · prompts.yaml

bangla_summarize_v3:
  name: bangla_summarize
  version: v3
  description: "Bangla article summarize, max-words constraint"
  template: |
    আপনি একজন বিশেষজ্ঞ Bangla editor। নিচের article-টি {{ max_words }}
    শব্দের মধ্যে summarize করুন। মূল ভাব ও key facts অপরিবর্তিত রাখুন।

    Article:
    {{ article }}

    Summary:

bangla_translate_v2:
  name: bangla_translate
  version: v2
  description: "English → Bangla, professional tone"
  template: |
    Translate the following English text to formal Bangla.
    Maintain technical terminology when appropriate.

    English:
    {{ text }}

    Bangla:

Self-documenting: name, version, description, template। Git diff prompt change clearly visible।

৫ · Versioning workflow

v1 production।
v2 develop in dev branch — edit prompt।
Regression suite run — eval against golden examples।
v2 PR review।
Shadow deploy — production prompt v1, log v2 hypothetical responses।
A/B test 5% traffic v2 — quality, cost, latency।
Promote v2 → production।
v1 archived।

৬ · Regression suite

Golden examples — known input/expected output। Prompt change-এ ensure no regression।

Python · Regression test

import pytest
import openai

GOLDEN = [
    {
        "input": {"article": "...", "max_words": 50},
        "expected_keywords": ["IT সেক্টর", "Bangladesh"],
        "expected_max_length": 60,  # words
    },
    # ... more
]

@pytest.mark.parametrize("case", GOLDEN)
def test_bangla_summarize_v3(case):
    prompt = render(bangla_summarize_v3, **case["input"])
    resp = openai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.0,
    )
    output = resp.choices[0].message.content

    # Length check
    assert len(output.split()) <= case["expected_max_length"]

    # Keyword presence
    for kw in case["expected_keywords"]:
        assert kw in output, f"Missing keyword: {kw}"

Temperature 0.0 — deterministic; flaky reduce। Golden suite-এ ১০-৫০ examples typical।

৭ · A/B testing prompts

Hash user_id → prompt v1 vs v2 assignment।
Track per-prompt metric: cost, latency, user satisfaction।
Statistical significance test (Lesson 23)।

৮ · Prompt registry tools

Langfuse: open-source, self-host or cloud; observability + prompt management।
PromptLayer: SaaS; prompt versioning + A/B।
Helicone: request logging + prompt tracking।
LangSmith: LangChain ecosystem; eval + tracing।

Prompt lifecycle — code-এর মতো formal lifecycle। Hardcoded → versioned + tested।

Bangladesh LLM team prompt sprawl common — 50+ prompts scattered scripts। Day-1 discipline cheap; refactor expensive।

ভাবনার প্রশ্ন

প্র ০১"Prompt sprawl — kibhabe prevent, kibhabe fix?"

Sprawl = prompts scattered codebase, no governance।

Prevention:

Day-1: prompts.yaml + version + name।
Code review checklist: "new LLM call — registered?"
CI lint: hardcoded prompt strings flagged।

Fix existing sprawl:

grep all openai.chat.create + "role": "user"।
Catalog every unique prompt।
Migrate to YAML — name + version।
Replace inline strings with template render call।

Governance:

Prompt owner per template।
Quarterly review — unused prompts archive।
Cost dashboard per prompt।

Anti-pattern:

Each engineer write own prompt — duplication।
Slight variation each → inconsistent quality।

মূল উপলব্ধি: Prompt = ML team's database query equivalent — needs schema, ownership, version। Sprawl normal but addressable. Discipline early avoid pain later।

প্র ০২"Rolling prompt change safely — কীভাবে?"

Prompt change subtle effect — sometimes invisible until specific input।

Safety steps:

Regression suite — golden examples।
Shadow — silent comparison।
Canary 5% traffic।
Monitor: cost, latency, user feedback।

What to monitor:

Output length (token cost)।
Hallucination rate (if measurable)।
User thumbs-up/down।
Conversion rate (if applicable)।

Rollback:

Feature flag — prompt version override।
Instant rollback via config update।

BD example — Bangla translation prompt:

v2 added "formal tone" instruction।
Shadow: 10% output significantly more formal।
Customer feedback: mixed — younger users prefer informal।
Result: keep v1, develop user-tier-aware prompt।

মূল উপলব্ধি: Prompt change hide subtle impact — code change-এর মতই rigorous validation। Skip-এ silent quality regression।

প্র ০৩"Prompt vs fine-tune — কখন?"

Both task adaptation; different mechanism + cost।

Prompt approach:

Pros: instant change, no training, easy A/B।
Cons: token cost (long prompt), context window limit, less consistent।

Fine-tune:

Pros: smaller prompt (cost ↓), more consistent, learn complex pattern।
Cons: training cost, retraining for change, less agile।

When prompt suffices:

Few-shot examples in prompt enough।
Task variation high — no consistent pattern।
Iteration needed — change prompt vs retraining।

When fine-tune justified:

Same task pattern → many examples।
Prompt too long → cost prohibitive।
Style/format consistency critical।
Smaller model (cost) acceptable quality।

BD context:

Bangla customer support tone — fine-tune justified (consistent style)।
Variable Q&A — prompt + RAG।
Hybrid: fine-tune base style, prompt for specifics।

মূল উপলব্ধি: Prompt = tactical, fast iteration। Fine-tune = strategic, locked behavior। Cost-volume crossover analyze।

প্র ০৪"Prompt-level metrics — কী track?"

Per-prompt visibility critical for cost + quality।

Cost metrics:

Average input tokens per call।
Average output tokens।
Cost per call (model rate × tokens)।
Daily/monthly cost per prompt।

Quality metrics:

User feedback (thumbs up/down)।
Hallucination rate (if detectable)।
Output schema match rate (structured output)।
Refusal rate (model says "I can't")।

Performance:

Latency p50/p99।
Token-per-second।
Streaming first-token-time।

Business:

Conversion rate (per-prompt-version)।
User retention (long-term)।

Tooling:

Langfuse, Helicone — purpose-built।
Custom Prometheus + tag।