পাঠ ২৯ · ৩২-এর মধ্যে · মডিউল ৪

Home / AI Courses / রিইনফোর্সমেন্ট লার্নিং / DPO ও RLAIF

DPO ও RLAIF — RLHF-এর সরলীকরণ

DPO & RLAIF: simplifying the RLHF pipeline

১০ মিনিট পড়া মাঝারি · Intermediate DPO loss in 5 lines

এই পাঠে যা শিখবেন

DPO-এর গাণিতিক derivation — কীভাবে RM eliminate হয়
DPO loss একটি single PyTorch function-এ
RLAIF — human-vs-AI feedback trade-off
Constitutional AI — principles + self-critique architecture

১ · RLHF-এর pain points

গত পাঠে আমরা দেখেছি — RLHF কাজ করে কিন্তু complicated। চারটি model একসাথে memory-তে — policy, reference, reward, value। PPO-এর instability — KL coefficient tune, advantage normalize, gradient clip। ৪-৫ engineering জোড়া হ্যান্ডস ফুল-টাইম।

প্রশ্ন: RM ও PPO দু'টোই কি অপরিহার্য? উত্তর: না।

DPO-র insight

Bradley-Terry preference model + KL-constrained reward maximization-এর closed-form solution আছে। সেই solution-এ implicit reward = $\beta \log(\pi_\theta / \pi_{\text{ref}})$। অর্থাৎ — policy নিজেই তার "secret RM"। Preferences থেকে policy directly শিখি, RM intermediate ছাড়াই।

ভাবুন আপনি বাজারে দাম জানতে চান। RLHF দু'ধাপে — (১) দাম-list বানান (RM), (২) সেই list দিয়ে কেনাকাটা (PPO)। DPO বলে — "list বানানো কেন? দু'টি জিনিস তুলনা করেই সরাসরি কিনুন।" Math একই, কাজ অনেক কম।

২ · DPO-র গাণিতিক derivation

RLHF objective:

$$\max_\pi \mathbb{E}_{x, y \sim \pi}[r(x,y)] - \beta \cdot \text{KL}(\pi \| \pi_{\text{ref}})$$

এই KL-regularized RL-এর closed-form optimal policy:

$$\pi^*(y|x) = \frac{1}{Z(x)} \pi_{\text{ref}}(y|x) \exp\left(\frac{r(x,y)}{\beta}\right)$$

Rearrange — reward as function of policy:

$$r(x,y) = \beta \log \frac{\pi^*(y|x)}{\pi_{\text{ref}}(y|x)} + \beta \log Z(x)$$

Bradley-Terry preference probability:

$$P(y_w \succ y_l | x) = \sigma(r(x, y_w) - r(x, y_l))$$

$\log Z(x)$ cancels (depends only on $x$)! তাই:

$$P(y_w \succ y_l | x) = \sigma\left(\beta \log \frac{\pi^*(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi^*(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right)$$

DPO loss — directly on preference data $(x, y_w, y_l)$:

$$\mathcal{L}_{\text{DPO}} = -\mathbb{E} \left[\log \sigma\left(\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right)\right]$$

Pure supervised loss। Policy gradient নাই, environment rollout নাই, RM নাই। শুধু $(x, y_w, y_l)$ batch — log-prob ratio compute, sigmoid, BCE।

DPO mathematically equivalent to RLHF when (a) Bradley-Terry preferences exact, (b) infinite data, (c) optimal RM trained। Practice-এ — DPO often slightly worse than well-tuned RLHF, but ৩-৫× cheaper, more stable। Trade-off worth it most use cases।

৩ · DPO PyTorch — পাঁচ লাইনে

Python · PyTorch · DPO loss

import torch, torch.nn.functional as F

def dpo_loss(policy, ref_policy, batch, beta=0.1):
    """
    batch = {
      'prompt_chosen_ids':   (B, T_c),
      'prompt_rejected_ids': (B, T_r),
      'prompt_chosen_mask':  (B, T_c),  # 1 over response, 0 over prompt+pad
      'prompt_rejected_mask':(B, T_r),
    }
    """
    def logprobs(model, ids, mask):
        out = model(ids).logits[:, :-1]                  # (B, T-1, V)
        target = ids[:, 1:]                              # (B, T-1)
        lp = F.log_softmax(out, -1).gather(-1, target.unsqueeze(-1)).squeeze(-1)
        # only over response tokens
        return (lp * mask[:, 1:]).sum(-1)                # (B,)

    pi_chosen   = logprobs(policy,     batch['prompt_chosen_ids'],   batch['prompt_chosen_mask'])
    pi_rejected = logprobs(policy,     batch['prompt_rejected_ids'], batch['prompt_rejected_mask'])
    with torch.no_grad():
        ref_chosen   = logprobs(ref_policy, batch['prompt_chosen_ids'],   batch['prompt_chosen_mask'])
        ref_rejected = logprobs(ref_policy, batch['prompt_rejected_ids'], batch['prompt_rejected_mask'])

    chosen_reward   = beta * (pi_chosen   - ref_chosen)
    rejected_reward = beta * (pi_rejected - ref_rejected)

    loss = -F.logsigmoid(chosen_reward - rejected_reward).mean()
    # diagnostics
    margin   = (chosen_reward - rejected_reward).mean().item()
    accuracy = (chosen_reward > rejected_reward).float().mean().item()
    return loss, {"margin": margin, "accuracy": accuracy}

পুরো DPO algorithm এতটুকু। Policy ও frozen reference দু'জনের log-prob ratio difference — Bradley-Terry binary cross-entropy। PPO-র ৫০০ লাইন code এখন ১৫ লাইন।

৪ · DPO-র শক্তি ও দুর্বলতা

শক্তি:

Simplicity: single supervised loss। Hyperparameter মাত্র $\beta$।
Stability: no rollouts, no reward exploitation cycle। Loss curve smooth।
Memory: ২ models (policy + ref), ৪ না। ২× memory save।
Data efficient: existing preference dataset reuse — UltraFeedback, HH-RLHF, Chatbot Arena।
Open-source friendly: Mistral, Llama-3, Gemma — সব DPO standard ship করে।

দুর্বলতা:

Offline only: data fixed। Online improvement-এ চাইলে — iterative DPO (sample new responses, re-label, retrain)।
Overoptimization: policy chosen response-এর likelihood বেশি বাড়ায়, rejected অনেক কমায় — but absolute likelihood-এ subtle issue। SimPO, IPO এই fix।
Reference dependency: $\pi_{\text{ref}}$ poor হলে — DPO খারাপ। Strong SFT essential।
Length bias: longer responses-এর log-prob বেশি — reward inflated। Length-normalized variants (SimPO)।

৫ · DPO variants — explosion of methods

IPO (Identity Preference Optimization): DPO-র overoptimization fix — sigmoid replace by squared loss। Theoretically grounded।
KTO (Kahneman-Tversky Optimization): "যেখানে preference নাই, শুধু binary thumbs-up/down — সেখানেও কাজ করে"। Practical।
cDPO: noisy labels হ্যান্ডল।
SimPO: reference-free, length-normalized। Llama-3-Instruct competitive।
ORPO: SFT + preference একসাথে — single-stage training।
NCA, RPO, BCO: বছরে নতুন variant। Field rapidly evolving।

৬ · RLAIF — humans replace করা

RLHF-এর সবচেয়ে costly bottleneck — human labelers। ১ million preference pairs ~$1M-$5M। Anthropic-এর "Constitutional AI" (২০২২) সমাধান দিল: AI labels itself।

RLAIF pipeline (Constitutional AI):

Step 1 — SL-CAI (Supervised Constitutional AI):
- Helpful-only model দিয়ে harmful prompts answer generate (red-teaming)।
- Same model প্রতিটি principle অনুযায়ী response critique।
- Same model revised, principle-aligned response generate।
- Original prompt + revised response → SFT fine-tune dataset।
Step 2 — RL-CAI:
- SL-CAI model দু'টি response generate per prompt।
- Same model (or stronger judge) — কোনটা principle-align — choose।
- Preference dataset → RM train → PPO/DPO।

"Constitution": ~৭০টি plain-English principles। যেমন:

"Choose the response that is least harmful and most helpful to the human."
"The response should not encourage illegal activity."
"Prefer responses that are honest about uncertainty."
"Avoid stereotyping based on race, gender, religion."

AI প্রতিটি principle-এর বিপরীতে response evaluate করে। Bangladeshi context-এর জন্য — local principles add (cultural sensitivity, religious harmony, language-specific norms)।

৭ · RLAIF-এর trade-offs

সুবিধা:

Cost: $0.001/comparison vs $1-10/human। ১০০০-১০,০০০× cheaper।
Speed: Parallel API calls — millions of labels per day।
Consistency: AI judge "একই" — inter-rater disagreement নেই।
Scaling: humans don't scale; AI does।
Iteration: principle update করে — instant relabel।

Risk:

Bias amplification: AI judge own biases label-এ encode। Human-AI gap গভীরায়।
Echo chamber: Same model labeling its own outputs — recursive blind spots।
Hard cases: nuanced ethical dilemmas — AI judge inconsistent।
Cultural blindness: AI trained on English-Western corpus — Bangla nuances miss।

তিনটি alignment recipe — RLHF (full pipeline), DPO (RM ও PPO eliminate), RLAIF (humans-ও eliminate)।

৮ · কখন কোনটি বাছবেন?

DPO: default choice ২০২৪+। Resource-constrained, open-source, fast iteration। Llama-3 fine-tune করতে চান — DPO।
RLHF/PPO: production-grade, multi-objective, online improvement, cost no-issue। Frontier labs।
RLAIF + Constitutional AI: safety-critical, scalable labeling, principle-driven। Anthropic Claude-এর secret sauce।
Hybrid: Modern best practice — SFT → DPO → RL polish + RLAIF data augmentation।

Field changing fast: ২০২৩-এ RLHF king, ২০২৪-এ DPO challenger, ২০২৫-এ likely SimPO/ORPO/new method। Foundation principle (Bradley-Terry preferences + KL regularization) স্থির — কিন্তু optimization technique evolve।

ভাবনার প্রশ্ন

প্রতিটি প্রশ্ন নিজে কিছুক্ষণ ভাবুন — তারপর "→ উত্তর" চাপুন।

প্র ০১ DPO mathematically RLHF-এর equivalent। তবু empirically RLHF কখনো কখনো better — কেন?

চমৎকার subtle question। Theory-practice gap-এর classic case।

(১) Online vs offline:

RLHF on-policy — current $\pi_\theta$ থেকে rollout, fresh feedback।
DPO offline — fixed dataset। Policy improve করলে — old preferences-এর coverage gap বাড়ে।
Solution: Iterative DPO — multiple rounds, between rounds new responses generate।

(২) Reward model expressivity:

Explicit RM separate from policy — different inductive bias। Sometimes better generalization।
DPO implicit RM = policy itself। Limited expressivity।

(৩) Overoptimization differently:

RLHF — reward hacks RM, KL keeps near SFT।
DPO — chosen log-prob increase, rejected decrease। But absolute log-prob can decrease overall (subtle issue)। Some work shows DPO model-এর likelihood SFT-এর চেয়ে কম — quality concern।

(৪) Distribution shift:

RLHF — policy stays near SFT due to KL। RM accurate near SFT।
DPO — same KL effect implicitly via reference, but no online correction।

(৫) Empirical evidence:

Original DPO paper (২০২৩): summarization, dialogue — DPO ≥ PPO।
Tulu 2 (২০২৩): DPO competitive with PPO।
Gemma technical report: DPO insufficient at scale, hybrid better।
Llama-3.1 technical report: extensive DPO + iterative + safety models।

(৬) Hyperparameter sensitivity:

RLHF more knobs but well-understood (PPO since ২০১৭)।
DPO simpler but $\beta$ critical। Wrong $\beta$ — collapse or no learning।

(৭) Production reality:

OpenAI/Anthropic — full RLHF + RLAIF, compute available।
Open-source — DPO economy choice।
Hybrid frontier: PPO fine-tune after DPO base — some labs use।

মূল উপলব্ধি: DPO 90% case-এ ভাল enough at 10% cost। Frontier-frontier (last 1% improvement) full RLHF apparatus pay off। Most users — DPO first, RLHF if needed।

প্র ০২ RLAIF "AI judges AI" — recursive blindspot তৈরি করার কথা। তবু Anthropic Claude এত safe কেন? কী differently?

চমৎকার alignment-philosophy question।

(১) Constitutional AI design:

Principles plain English-এ written — humans audit-able।
AI judge নিজে generate করে না principles; humans curate।
"Helpful + Harmless + Honest" trade-off explicit।

(২) Stronger judge model:

Judge ≠ policy। Often stronger model (Claude-3-Opus judges Claude-3-Sonnet)।
Asymmetry breaks recursive trap।

(৩) Multi-stage human oversight:

Constitution itself written by humans (Anthropic researchers + iterations)।
Spot-check sampling — random AI labels human-verify।
Red-team ongoing — adversarial prompts, jailbreaks identified।
Post-deployment monitoring — user reports trigger investigation।

(৪) Different blind spots:

Humans + AI mistakes don't perfectly correlate।
Mixing human + AI labels — coverage union of error patterns।
"AI vouches" + "Human spot-check" — both required।

(৫) Specific Constitutional AI tricks:

Multiple critiques per response: AI critique against multiple principles, sample one।
Chain-of-thought: AI explains reasoning before judgment। Auditable।
Adversarial pairing: intentionally diverse responses (one harmful, one safe) — AI must distinguish।

(৬) Bias mitigation:

Diverse training corpus → judge less biased to single perspective।
Specifically anti-bias principles in constitution।
Bias evaluation suite (BBQ, BOLD) regular monitoring।

(৭) Failure modes acknowledged:

Anthropic-এর own research: Claude সখানে fails — sycophancy, evasion, hidden harms।
Continuous improvement, not "solved"।
Mechanistic interpretability research — circuit-level safety understanding।

(৮) Comparative safety:

External evals (HHEM, MT-Bench-Safety, RewardBench) — Claude top-3 consistently।
Jailbreak resistance highest among major models (২০২৪ data)।
Hallucination rate competitive।

(৯) Open question: "AI-AI alignment recursive eventually fail?" — research অযৌক্তিক না। Mesa-optimization, deceptive alignment — long-term concerns। Constitutional AI buys time, not final solution।

মূল কথা: RLAIF safe হলে — humans-in-the-loop at higher abstraction level (constitution writing, oversight)। "Pure AI judge" naïve; "AI + human + structured principles" robust। Engineering layered defense, not single magic technique।

প্র ০৩ $\beta$ DPO-তে RLHF-এর KL coefficient-এর মতো — বড়/ছোট হলে কী হয়? Tuning intuition?

চমৎকার practical hyperparameter question।

$\beta$ এর role — KL regularization strength:

Implicit reward $r = \beta \log(\pi/\pi_{\text{ref}})$।
Large $\beta$: small log-ratio difference → large reward difference। Policy stays near reference।
Small $\beta$: large log-ratio difference → small reward difference। Policy can move far।

$\beta$ very small (e.g., 0.01):

Reward range narrow — preferences encode কম।
Loss easily saturate (sigmoid extreme tail)।
Policy moves a lot from reference — reference irrelevant।
Risk: degenerate to maximizing chosen log-prob alone, ignore KL anchor।
Empirical: training unstable, accuracy plateau low।

$\beta$ very large (e.g., 1.0):

Tiny policy change → big reward shift।
Loss easily saturate other direction (stuck near reference)।
Improvement minimal।
Empirical: model behavior similar to SFT, no gain।

Sweet spot ($\beta = 0.1$ - $0.3$ typically):

Original DPO paper: $\beta=0.1$ default।
Anthropic-style preference: $\beta=0.2-0.5$।
Domain-specific: longer responses → smaller $\beta$ (length factor)।

Tuning intuition:

Start $\beta=0.1$।
Track validation accuracy + reward margin।
Margin > 5: probably overoptimizing, increase $\beta$।
Margin < 1: not learning enough, decrease $\beta$।
Eval downstream task — ultimate truth।

Comparison to RLHF KL:

RLHF: explicit KL term, tunable runtime। Adaptive controllers possible।
DPO: $\beta$ fixed at training time। Less flexible, simpler।

Pathological cases:

Reference too weak — DPO struggles regardless of $\beta$।
Preferences low-quality — $\beta$ tuning won't fix data issue।
Length bias — even $\beta$ tuning insufficient; SimPO needed।

Practical advice:

Default $\beta=0.1$, sweep $\{0.05, 0.1, 0.2, 0.5\}$।
Monitor "rewards/margins" and "rewards/accuracies" — TRL library exposes।
Validate on held-out preference set + downstream eval (MT-Bench)।

মূল কথা: $\beta$ — DPO-এর single most-important knob। Wrong $\beta$ = wasted training run।

প্র ০৪ আপনি Bangla কোম্পানি — ১০টি GPU, $50K বাজেট, ৩-মাস timeline-এ একটি Bangla customer-support agent বানাতে হবে। DPO-RLHF-RLAIF-এর কোন combination?

চমৎকার applied resource-constrained scenario।

Initial approach — তিন-tier strategy:

Foundation: Llama-3.1-8B base বা Aya-23-8B (multilingual)।
Stage 1 — Bangla SFT (১ মাস, $20K):
- ~৫০K Bangla customer-support dialogues collect।
- Sources: existing chat logs (anonymized), translation from English support data, synthetic from GPT-4।
- LoRA fine-tune (rank 32) — full param-tune করতে গেলে compute চাই।
- 4 GPUs × 2 weeks = ৬৪ GPU-day।
Stage 2 — RLAIF preference data (২ সপ্তাহ, $5K):
- Constitution লিখুন — Bangla-specific (10-15 principles)।
- SFT model generate ২ responses per prompt, ~৩০K prompts।
- GPT-4 + Claude judge — Bangla-aware। Cost ~$3-5K।
- Humans (Bangladesh annotators) ১,০০০ comparisons spot-check + correct ($2K)।
Stage 3 — DPO training (৩ সপ্তাহ, $15K):
- DPO loss + LoRA, 4 GPUs × 2 weeks।
- Iterative — initial DPO → new responses generate → re-label → second DPO round।
- Hyperparam: $\beta=0.1$ start, sweep ৩টি।
Stage 4 — Eval + iteration (২ সপ্তাহ, $10K):
- Bangla customer-support test suite — 500 prompts।
- Human eval ($5K) ও automated (BLEU, BERTScore, GPT-4 judge)।
- A/B test internal — current rule-based vs LLM।
- Safety red-teaming — Bangla jailbreak attempts।

Why this combination:

DPO over PPO: $50K budget — PPO infrastructure prohibitive। DPO 5× cheaper।
RLAIF for scale: ১০K human comparisons would cost $30K alone — RLAIF brings it to $3K।
LoRA over full fine-tune: 8B model full FT needs 80GB+ per GPU। LoRA 24GB doable on standard GPU।
Hybrid human+AI labels: pure RLAIF risky for production; humans validate critical principles।

Bangla-specific considerations:

Tokenizer: base model-এর tokenizer Bangla-friendly check। Llama-3 OK; older models bloat।
Code-switching: "ami order ta cancel korte chai" — 50%+ customer queries। Train data must include।
Address forms: Auto-detect formal (আপনি) vs casual (তুমি) — match user's tone।
Religious sensitivity: festivals, dietary preferences — politely accommodate।
Localization: currency BDT, time zones GMT+6, addresses Bangladeshi format।

Risks ও mitigation:

Hallucination: wrong order info = customer angry। Solution: RAG with order DB, "I'll check" fallback।
Escalation: AI uncertain → human agent। Confidence threshold।
Privacy: customer data — local hosting, GDPR-style protection।
Compliance: Bangladesh Telecom Act/Consumer Rights Act — refund policies in answer cap।

Deliverables timeline:

Week 4: SFT model alpha — internal testing।
Week 7: DPO model beta — pilot with select customers।
Week 10: Production-ready, monitoring dashboard।
Week 12: Public launch + continuous improvement framework।

Continuous improvement (post-launch):

User feedback loops — thumbs up/down → preference data।
Monthly DPO refresh with new data।
Quarterly constitution update — issues addressed।

মূল কথা: $50K + ১০ GPU + ৩ মাস — Bangladesh-এ realistic budget। DPO + RLAIF + LoRA — sweet spot। Production deploy + iterate, course-end project hিsebe excellent।

অনুশীলন

DPO loss derivation: RLHF objective থেকে DPO loss-এ পৌঁছানোর key step কী? কোন term cancel হয়?

(১) KL-regularized RL-এর closed-form optimal $\pi^* \propto \pi_{\text{ref}} \exp(r/\beta)$। (২) Reward as $r = \beta \log(\pi^*/\pi_{\text{ref}}) + \beta \log Z$। (৩) Bradley-Terry: $\sigma(r_w - r_l)$। (৪) $\log Z(x)$ both terms-এ একই — cancel। তাই DPO loss = $-\log\sigma(\beta \log(\pi/\pi_{\text{ref}})$-এর difference)।
RLAIF cost-benefit: ১০০K preference labels — humans-এ $500K, RLAIF-এ $500। কী কী scenarios-এ "human-only" তবু worth?

(১) Cultural/legal nuance — Bangladesh-specific religious sensitivity AI-judge miss। (২) Frontier model alignment — AI-judge model itself untrustworthy। (৩) Safety-critical — medical, legal advice। (৪) Subjective taste — humor, creativity। (৫) Bias-evaluation — must come from humans। Hybrid (90% RLAIF + 10% human spot-check) best practice।
Method choice: (a) ৫০K Llama-3 fine-tune budget, (b) $5M Claude-equivalent build, (c) Quick research prototype। কোন approach বাছবেন?

(a) ৫০K — DPO + LoRA। PPO infeasible। (b) $5M — full RLHF + RLAIF + Constitutional AI hybrid। Multi-stage iterative। (c) Prototype — DPO with HuggingFace TRL library, 1 day setup, default hyperparams। Validate idea cheaply, scale if needed।