পাঠ ২৯ · ৩৩-এর মধ্যে · মডিউল ৪

Home / AI Courses / MLOps / Prompt management

Prompt management ও versioning

Prompts as code — version, test, deploy

৭ মিনিট পড়া উচ্চ · Advanced Concept

এই পাঠে যা শিখবেন

Prompt-এর storage strategies
Templating tools
Versioning + A/B testing
Regression suite

১ · "Prompt = code" mindset

Prompt code-এর মতো production artifact. Hardcoded string-এ বসিয়ে দিলে — change track নেই, A/B test নেই, regression catch হবে না।

২ · Storage approaches

Git-tracked files: simplest; .txt or .yaml in repo. Pros: Git workflow. Cons: deploy required for change.
Database: dynamic update; versioned. Pros: hot-reload. Cons: schema, governance.
Prompt registry SaaS: Langfuse, PromptLayer, Helicone — purpose-built. Pros: UI, A/B built-in. Cons: cost.

৩ · Templating

Python · Prompt template

from jinja2 import Template
from dataclasses import dataclass
import yaml

@dataclass
class PromptVersion:
    name: str
    version: str
    template: str
    description: str

# Load from YAML — git-tracked
PROMPTS = yaml.safe_load(open("prompts.yaml"))

bangla_summarize = PromptVersion(**PROMPTS["bangla_summarize_v3"])

def render(prompt: PromptVersion, **vars) -> str:
    return Template(prompt.template).render(**vars)

# Usage
prompt_text = render(
    bangla_summarize,
    article="Bangladesh-এর IT সেক্টর এখন...",
    max_words=100,
)

# Track in logging
import openai
resp = openai.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": prompt_text}],
)

logger.info({
    "prompt_name": bangla_summarize.name,
    "prompt_version": bangla_summarize.version,
    "input_tokens": resp.usage.prompt_tokens,
    "output_tokens": resp.usage.completion_tokens,
})

Pattern: prompts YAML, version field, log version + tokens. Production tracing connect specific prompt-version-এ।

৪ · prompts.yaml example

YAML · prompts.yaml

bangla_summarize_v3:
  name: bangla_summarize
  version: v3
  description: "Bangla article summarize, max-words constraint"
  template: |
    আপনি একজন বিশেষজ্ঞ Bangla editor. নিচের article-টি {{ max_words }}
    শব্দের মধ্যে summarize করুন। মূল ভাব ও key facts অপরিবর্তিত রাখুন।

    Article:
    {{ article }}

    Summary:

bangla_translate_v2:
  name: bangla_translate
  version: v2
  description: "English → Bangla, professional tone"
  template: |
    Translate the following English text to formal Bangla.
    Maintain technical terminology when appropriate.

    English:
    {{ text }}

    Bangla:

Self-documenting: name, version, description, template. Git diff prompt change clearly visible.

৫ · Versioning workflow

v1 production.
v2 develop in dev branch — edit prompt.
Regression suite run — eval against golden examples.
v2 PR review.
Shadow deploy — production prompt v1, log v2 hypothetical responses.
A/B test 5% traffic v2 — quality, cost, latency.
Promote v2 → production.
v1 archived.

৬ · Regression suite

Golden examples — known input/expected output. Prompt change-এ ensure no regression.

Python · Regression test

import pytest
import openai

GOLDEN = [
    {
        "input": {"article": "...", "max_words": 50},
        "expected_keywords": ["IT সেক্টর", "Bangladesh"],
        "expected_max_length": 60,  # words
    },
    # ... more
]

@pytest.mark.parametrize("case", GOLDEN)
def test_bangla_summarize_v3(case):
    prompt = render(bangla_summarize_v3, **case["input"])
    resp = openai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.0,
    )
    output = resp.choices[0].message.content

    # Length check
    assert len(output.split()) <= case["expected_max_length"]

    # Keyword presence
    for kw in case["expected_keywords"]:
        assert kw in output, f"Missing keyword: {kw}"

Temperature 0.0 — deterministic; flaky reduce. Golden suite-এ ১০-৫০ examples typical.

৭ · A/B testing prompts

Hash user_id → prompt v1 vs v2 assignment.
Track per-prompt metric: cost, latency, user satisfaction.
Statistical significance test (Lesson 23).

৮ · Prompt registry tools

Langfuse: open-source, self-host or cloud; observability + prompt management.
PromptLayer: SaaS; prompt versioning + A/B.
Helicone: request logging + prompt tracking.
LangSmith: LangChain ecosystem; eval + tracing.

Prompt lifecycle — code-এর মতো formal lifecycle. Hardcoded → versioned + tested.

Bangladesh LLM team prompt sprawl common — 50+ prompts scattered scripts. Day-1 discipline cheap; refactor expensive.

ভাবনার প্রশ্ন

প্র ০১"Prompt sprawl — kibhabe prevent, kibhabe fix?"

Sprawl = prompts scattered codebase, no governance.

Prevention:

Day-1: prompts.yaml + version + name.
Code review checklist: "new LLM call — registered?"
CI lint: hardcoded prompt strings flagged.

Fix existing sprawl:

grep all openai.chat.create + "role": "user".
Catalog every unique prompt.
Migrate to YAML — name + version.
Replace inline strings with template render call.

Governance:

Prompt owner per template.
Quarterly review — unused prompts archive.
Cost dashboard per prompt.

Anti-pattern:

Each engineer write own prompt — duplication.
Slight variation each → inconsistent quality.

মূল উপলব্ধি: Prompt = ML team's database query equivalent — needs schema, ownership, version. Sprawl normal but addressable. Discipline early avoid pain later.

প্র ০২"Rolling prompt change safely — কীভাবে?"

Prompt change subtle effect — sometimes invisible until specific input.

Safety steps:

Regression suite — golden examples.
Shadow — silent comparison.
Canary 5% traffic.
Monitor: cost, latency, user feedback.

What to monitor:

Output length (token cost).
Hallucination rate (if measurable).
User thumbs-up/down.
Conversion rate (if applicable).

Rollback:

Feature flag — prompt version override.
Instant rollback via config update.

BD example — Bangla translation prompt:

v2 added "formal tone" instruction.
Shadow: 10% output significantly more formal.
Customer feedback: mixed — younger users prefer informal.
Result: keep v1, develop user-tier-aware prompt.

মূল উপলব্ধি: Prompt change hide subtle impact — code change-এর মতই rigorous validation. Skip-এ silent quality regression.

প্র ০৩"Prompt vs fine-tune — কখন?"

Both task adaptation; different mechanism + cost.

Prompt approach:

Pros: instant change, no training, easy A/B.
Cons: token cost (long prompt), context window limit, less consistent.

Fine-tune:

Pros: smaller prompt (cost ↓), more consistent, learn complex pattern.
Cons: training cost, retraining for change, less agile.

When prompt suffices:

Few-shot examples in prompt enough.
Task variation high — no consistent pattern.
Iteration needed — change prompt vs retraining.

When fine-tune justified:

Same task pattern → many examples.
Prompt too long → cost prohibitive.
Style/format consistency critical.
Smaller model (cost) acceptable quality.

BD context:

Bangla customer support tone — fine-tune justified (consistent style).
Variable Q&A — prompt + RAG.
Hybrid: fine-tune base style, prompt for specifics.

মূল উপলব্ধি: Prompt = tactical, fast iteration. Fine-tune = strategic, locked behavior. Cost-volume crossover analyze.

প্র ০৪"Prompt-level metrics — কী track?"

Per-prompt visibility critical for cost + quality.

Cost metrics:

Average input tokens per call.
Average output tokens.
Cost per call (model rate × tokens).
Daily/monthly cost per prompt.

Quality metrics:

User feedback (thumbs up/down).
Hallucination rate (if detectable).
Output schema match rate (structured output).
Refusal rate (model says "I can't").

Performance:

Latency p50/p99.
Token-per-second.
Streaming first-token-time.

Business:

Conversion rate (per-prompt-version).
User retention (long-term).

Tooling:

Langfuse, Helicone — purpose-built.
Custom Prometheus + tag.