পাঠ ১৭ · ২৮-এর মধ্যে · মডিউল ৩

Home / AI Courses / Generative AI / Classifier-free guidance

Classifier-free guidance (CFG)

Classifier-free guidance — boosting prompt adherence

৬ মিনিট পড়া মাঝারি · Intermediate guidance_scale=7.5

এই পাঠে যা শিখবেন

Classifier guidance (Dhariwal & Nichol 2021) — পূর্বের approach
Classifier-free guidance (Ho & Salimans 2022) — কেন superior
Guidance scale-এর effect, optimal range, artifacts
Negative prompt — same idea-র creative use

১ · Conditional generation challenge

Diffusion-এ text condition $c$ যোগ করতে — $\epsilon_\theta(\mathbf{x}_t, t, c)$ train করা হয় (cross-attention)। কিন্তু সমস্যা — generated sample condition-কে loosely follow করে। Diversity-র জন্য network কম attention দেয় text-এ।

উদাহরণ: "a red car under blue sky" → SD often "blue car under red sky" বা ignore করে।

২ · Classifier guidance — first attempt

Dhariwal & Nichol (2021, "Diffusion Models Beat GANs") — separate classifier $p(c \mid \mathbf{x}_t)$ train করুন। Score modify করুন:

$$\hat\epsilon(\mathbf{x}_t, t, c) = \epsilon_\theta(\mathbf{x}_t, t) - w\sqrt{1-\bar\alpha_t}\,\nabla_{\mathbf{x}_t} \log p(c \mid \mathbf{x}_t)$$

Bayes: $\nabla \log p(\mathbf{x}_t \mid c) = \nabla \log p(\mathbf{x}_t) + \nabla \log p(c \mid \mathbf{x}_t)$।

সমস্যা:

Separate classifier train করতে হয় — extra compute, dataset।
Classifier noisy data-তে train — fragile, adversarial।
Free-form text condition-এর জন্য infeasible।

৩ · Classifier-free guidance — Ho & Salimans 2022

Brilliant simplification: এক network-এ conditional ও unconditional একসাথে শেখান।

Training:

Random ১০-২০% sample-এ condition $c$ replace with null token $\emptyset$।
Network learns both $\epsilon_\theta(\mathbf{x}_t, t, c)$ এবং $\epsilon_\theta(\mathbf{x}_t, t, \emptyset)$।
Same network capacity, no extra cost।

Sampling:

$$\hat\epsilon = \epsilon_\theta(\mathbf{x}_t, t, \emptyset) + w \big(\epsilon_\theta(\mathbf{x}_t, t, c) - \epsilon_\theta(\mathbf{x}_t, t, \emptyset)\big)$$

$w$ = guidance scale (typically $1$-$15$, default ~$7.5$)।

Geometric intuition

Conditional ε ও unconditional ε-এর পার্থক্য — "condition যোগ করায় network কোন direction-এ আরো push করে"। CFG সেই direction-কে $w$ গুণ বাড়িয়ে নেয় — extrapolation। $w=1$ → conditional only। $w > 1$ → over-conditioning। $w=0$ → unconditional।

৪ · Sampling cost — 2× per step

প্রতি step-এ U-Net দু'বার call: একবার $c$, একবার $\emptyset$। Practical-এ batch-এ একসাথে — $2B$ inputs। Memory ও compute roughly 2×।

১৫-২০% সাধ্য বৃদ্ধির বিনিময়ে — major quality boost। সব production diffusion CFG ব্যবহার করে।

৫ · Guidance scale-এর effect

CFG noise space-এ extrapolation। $w$ বাড়ালে conditional direction-এ আরো push — কিন্তু একটি limit-এর পরে artifacts বাড়ে।

$w = 0$: Pure unconditional — random ছবি, prompt ignored।
$w = 1$: Standard conditional — weak prompt adherence।
$w = 3-5$: Balanced — natural, prompt-aware।
$w = 7-8$: Stable Diffusion default — strong adherence, slight saturation।
$w = 15+$: Over-saturated, artifacts, "AI-generated" look।

৬ · Negative prompts

Brilliant CFG extension: $\epsilon_\theta(\mathbf{x}_t, t, \emptyset)$-এর জায়গায় $\epsilon_\theta(\mathbf{x}_t, t, c_{neg})$ ব্যবহার করুন।

$$\hat\epsilon = \epsilon_\theta(\mathbf{x}_t, t, c_{neg}) + w\big(\epsilon_\theta(\mathbf{x}_t, t, c_{pos}) - \epsilon_\theta(\mathbf{x}_t, t, c_{neg})\big)$$

Result: image $c_{pos}$-এর দিকে যায়, $c_{neg}$ থেকে দূরে।

Practical use: "blurry, low quality, distorted, watermark, text" — quality boost-এর জন্য standard negative prompt।

৭ · PyTorch implementation

Python · PyTorch

import torch

# Training: random condition dropout
def train_step_with_dropout(model, x0, c, dropout_prob=0.1):
    B = x0.shape[0]
    t = torch.randint(0, 1000, (B,), device=x0.device)
    noise = torch.randn_like(x0)
    xt = q_sample(x0, t, noise)

    # 10% chance to replace c with null embedding
    mask = torch.rand(B, device=x0.device) > dropout_prob
    null_emb = torch.zeros_like(c)              # or learned ∅ embedding
    c_in = torch.where(mask.view(-1, 1, 1), c, null_emb)

    pred = model(xt, t, c_in)
    return F.mse_loss(pred, noise)

# Sampling with CFG
@torch.no_grad()
def cfg_sample(model, c, c_neg=None, w=7.5, T=50, shape=(1, 4, 64, 64)):
    if c_neg is None:
        c_neg = torch.zeros_like(c)             # null condition

    x = torch.randn(shape, device=c.device)
    for t in reversed(range(T)):
        # batch the two passes for efficiency
        x_in = torch.cat([x, x], dim=0)
        c_in = torch.cat([c_neg, c], dim=0)
        eps_both = model(x_in, t, c_in)
        eps_neg, eps_pos = eps_both.chunk(2)

        # CFG combination
        eps = eps_neg + w * (eps_pos - eps_neg)

        # standard denoising step
        x = denoise_step(x, eps, t)
    return x

Stable Diffusion-এর সব pipeline এই pattern-এ চলে — batch-এ two pass একসাথে। Negative prompt simply c_neg-এ inject।

৮ · Recent variants

Dynamic CFG: $w$ varies with $t$ — early high, late low।
CFG++ (Chung et al. 2024): Adaptive guidance, fewer artifacts।
Perp-Neg: Negative prompt orthogonal projection — more controllable।
Distilled CFG: Distill into single-pass model — 2× speedup। SDXL Turbo, LCM use this।

Common SD problem: high CFG ($w \ge 12$) → "burned" oversaturated colors, plastic skin। Solutions: lower $w$, dynamic threshold (Imagen), or rescaling.

ভাবনার প্রশ্ন

প্রতিটি প্রশ্ন নিজে কিছুক্ষণ ভাবুন — তারপর "→ উত্তর" চাপুন।

প্র ০১ CFG কেন গাণিতিকভাবে interpretable — score-based view-এ কী ঘটছে? কেন এটি Bayes rule-এর "implicit classifier"-এর সমতুল্য?

CFG সাদামাটা-দেখানো trick — আসলে গভীর Bayesian semantics আছে।

Score-based view:

$\epsilon_\theta(\mathbf{x}_t, t, c) \propto -\nabla_{\mathbf{x}_t} \log p_t(\mathbf{x}_t \mid c)$ (conditional score)।
$\epsilon_\theta(\mathbf{x}_t, t, \emptyset) \propto -\nabla_{\mathbf{x}_t} \log p_t(\mathbf{x}_t)$ (unconditional score)।
Difference: $-(\epsilon_{cond} - \epsilon_{uncond}) \propto \nabla \log p_t(\mathbf{x}_t \mid c) - \nabla \log p_t(\mathbf{x}_t)$।
Bayes: $\log p(\mathbf{x} \mid c) - \log p(\mathbf{x}) = \log p(c \mid \mathbf{x}) - \log p(c)$।
$\log p(c)$ has no $\mathbf{x}$-dependence, gradient = 0।
So: $\nabla \log p(\mathbf{x} \mid c) - \nabla \log p(\mathbf{x}) = \nabla \log p(c \mid \mathbf{x})$।

Implication:

$(\epsilon_{cond} - \epsilon_{uncond})$ ≈ negative gradient of "implicit classifier" $p(c \mid \mathbf{x}_t)$।
CFG: $\hat\epsilon = \epsilon_{uncond} + w(\epsilon_{cond} - \epsilon_{uncond})$।
Equivalent to sampling from $p(\mathbf{x} \mid c)^w \cdot p(\mathbf{x})^{1-w}$ — "tempered" posterior।
$w > 1$: posterior sharpened (more confident classification)।

Comparison with classifier guidance:

Classifier: explicit $p(c \mid \mathbf{x}_t)$ network — separate training।
CFG: same network does both — implicit classifier free।
"Classifier-free" name comes from this: no separate classifier needed।

Why "free" is misleading:

Sampling 2× compute (two model passes)।
"Free" = no extra network, but sampling cost real।
Modern distillation (Turbo, LCM) addresses this।

Bayesian temperature interpretation:

$p_w(\mathbf{x} \mid c) \propto p(\mathbf{x} \mid c) \cdot p(c \mid \mathbf{x})^{w-1}$ — "amplified evidence"।
$w \to 0$: prior $p(\mathbf{x})$।
$w = 1$: posterior $p(\mathbf{x} \mid c)$।
$w \to \infty$: argmax $\arg\max_{\mathbf{x}} p(c \mid \mathbf{x})$ — most "classifier-pleasing" image।

Why amplification beneficial:

$p(c \mid \mathbf{x})$ generally weak — CLIP text-image similarity not perfectly calibrated।
Amplification mode-seeks — image strongly conforming to text condition।
Trade: diversity (broad posterior) vs precision (sharp posterior)।

Failure modes:

$w$ too high — sample exit data manifold. Pixels saturated, off-distribution textures।
"Burning" — CFG amplifies high-frequency texture noise।
Solution: dynamic thresholding (Imagen) — pixel value rescaling।

Modern theoretical refinements:

CFG++ (Chung 2024): geometric reformulation, no oversaturation।
Adaptive guidance: $w$ varies based on $\|\epsilon_{cond} - \epsilon_{uncond}\|$।
Perpendicular guidance: project orthogonal to data manifold।

Connection to RLHF:

Reward-tilted sampling: $p_\beta(\mathbf{x}) \propto p(\mathbf{x}) e^{\beta r(\mathbf{x})}$।
CFG: similar amplification with implicit reward $\log p(c \mid \mathbf{x})$।
Diffusion-RLHF: explicit reward + diffusion sampling = aligned generation।

মূল উপলব্ধি: CFG looks like a hack — কিন্তু Bayesian "tempered posterior" sampling-এর elegant approximation। "Magic 7.5" empirical, but underlying math principled। এই insight modern alignment, RLHF for diffusion-এর ভিত্তি।

প্র ০২ "Default $w = 7.5$" — কেন এই magic number? Domain (face, landscape, art), resolution, model size — কোনটি optimal $w$ change করে?

Magic 7.5 — Stable Diffusion-এ Hugging Face default। কিন্তু optimal value context-specific।

Why 7.5 emerged:

Original DALL·E 2 paper used "guidance scale 4" with their formulation।
Imagen used 7.5 — strong text-following at $1024^2$।
SD adopted Imagen's 7.5।
Empirical sweet spot — most prompts work, mild oversaturation tolerable।

Domain dependence:

Photorealistic (people, landscape): $w = 5-7.5$। Higher creates plastic look।
Anime/illustration: $w = 8-12$। Saturated colors aesthetically OK।
Faces (portrait): $w = 4-6$। Skin texture preservation।
Abstract art: $w = 3-5$। Diversity preferred।
Text rendering: $w = 8-12$। Strong adherence necessary।

Resolution dependence:

$512^2$ (SD1.5): $w \approx 7.5$।
$1024^2$ (SDXL): $w \approx 5-7$ (less needed due to bigger model capacity)।
$2048+$ upscale: $w$ effectively reduced (per-pixel guidance dilutes)।

Model size dependence:

Smaller model (SD1.5 860M): higher $w$ needed for adherence।
Larger model (SD3 2B+): lower $w$ adequate — better intrinsic alignment।
Massive (Flux 12B): often $w = 3.5$ suffices।
Pattern: bigger → less guidance needed।

Sampler interaction:

DDIM, Euler: $w$ standard range 5-10।
DPM-Solver++: tolerates higher $w$ (10-12) due to multi-step error correction।
LCM/Turbo: $w \approx 1-3$ (distillation absorbs guidance)।
Few-step: high $w$ catastrophic — $w \le 2$।

Step count interaction:

20 step + $w=7.5$: artifacts more pronounced।
50 step + $w=7.5$: cleaner, $w$ amplification distributed।
100 step + $w=10$: sometimes acceptable।

Negative prompt interaction:

Strong negative prompt + high $w$ → over-correction artifacts।
Mild negative + $w=7.5$ → balanced।
"Embedding" negative prompts (Embedding Inversion) → effective at lower $w$।

Dynamic CFG (advanced):

$w(t) = w_{base} \cdot f(t)$।
Common: high early ($t$ near $T$), low late (near $t=0$)। Structure determined early, fine details later।
Imagen, GLIDE used variants।

Empirical search:

For new model: grid search $w \in \{3, 5, 7.5, 10, 12\}$ — pick visually best।
Dataset-level eval: generate 100 images per $w$, FID + CLIP score।
Per-prompt: complex prompts need higher; simple prompts lower।

Practical recommendations:

SD1.5 photo: 5-7।
SDXL photo: 4-6।
Flux: 3-4।
Anime models: 7-12।
Always experiment per-domain।

Common mistake:

"Increase CFG until it follows prompt" — leads to artifacts। Better: improve prompt, or use ControlNet।
CFG ≠ prompt strength alone — model capacity, training also matters।

মূল উপলব্ধি: $w = 7.5$ historical default; optimal context-dependent। Modern models trend toward lower $w$ (3-5) due to better intrinsic alignment। Practitioners-এর জন্য systematic experimentation > magic number।

প্র ০৩ Negative prompt — same CFG mechanism দিয়ে inverse direction। Common patterns ("blurry, low quality"), Textual Inversion-based negatives, embedding-based "BadHands" — কোনটি কখন best? Effective negative prompt কী principle-এ build করব?

Negative prompt — SD-এর "secret weapon"। প্রায়ই positive prompt-এর চেয়ে impactful।

Mechanism:

Standard CFG: $\epsilon_{uncond} + w(\epsilon_{cond} - \epsilon_{uncond})$।
Replace $\epsilon_{uncond}$ with $\epsilon_{cond_{neg}}$।
Image moves from $c_{neg}$ direction toward $c_{pos}$ direction।
Implicit "stay away from $c_{neg}$"।

Common negative prompt patterns:

Quality boost: "blurry, low quality, low resolution, jpeg artifacts, watermark, text, signature"।
Anatomy: "deformed, distorted, malformed, disfigured, bad anatomy, extra limbs, missing fingers, fused hands"।
Style avoidance: "cartoon, anime, painting" (for photo); "photo, realistic" (for anime)।
Composition: "cropped, out of frame, cut off"।
Color: "oversaturated, desaturated, muted"।

SDXL standard negative:

"low quality, worst quality, normal quality, lowres, blurry, jpeg artifacts" — community baseline।

Anime model negative (Civitai):

"easynegative" — Textual Inversion embedding (single token captures multiple bad attributes)।
"badhandv4" — for hand anatomy।
"ng_deepnegative_v1" — generic quality।
Embedding negatives more compact, more effective than text।

Textual Inversion negative embeddings:

Train embedding on bad images → use as negative।
"BadHands" — trained on malformed hand images।
Single token, captures complex visual concept।
More effective than verbal description।

Effective negative prompt principles:

Specific over generic: "extra fingers" beats "bad anatomy"।
Visual concrete: Words SD/CLIP recognizes — avoid abstract।
Don't contradict positive: "red car" + "no red" → confused।
Match domain: Photo negative ≠ anime negative।
Less is more: Long negative dilutes effect; 5-15 keywords optimal।

Failure modes:

Over-negative: "ugly, deformed, terrible, awful, horrible, nasty, gross" — diminishing return, may hurt diversity।
Style erasure: "anime" in negative for photo → kills natural manga-influenced compositions।
Concept rebound: "no cat" sometimes generates cat (negation poor in CLIP)।

Advanced techniques:

Perp-Neg (Armandpour 2023): Project negative orthogonal to positive — cleaner separation।
Per-region negative: Spatial mask + region-specific negative।
Time-varying negative: Strong early, fade out — preserve detail।
Negative prompt LoRA: Train LoRA on bad outputs, apply with negative weight।

Domain-specific negatives:

Portrait: "asymmetric eyes, malformed mouth, wrinkled, age spots, skin defects"।
Architecture: "wonky perspective, melting walls, distorted geometry"।
Product photography: "blurry product, watermark, retail tag, shadow distortion"।
Bangladeshi cultural: "Indian, Pakistani, generic Asian, Western, stereotypical"।

Modern model considerations:

SD3, Flux: better intrinsic quality — minimal negative needed।
SD1.5 community: heavy negative essential।
Distilled models (Turbo, Schnell): negative often ineffective (fast inference can't apply CFG fully)।

Iteration workflow:

Start without negative। Identify recurring issues।
Add specific terms for those issues।
A/B test with/without each term।
Save effective negative as preset।

Privacy/safety negatives:

"NSFW, nudity, explicit" — for safe production environments।
"Children, kids" — for adult-oriented content (responsible use)।
"Real person, celebrity" — to avoid identity issues।

মূল উপলব্ধি: Negative prompt = "implicit data filter" within CFG framework। Quality often defined by what you exclude, not just what you include। Designer/prompt engineer-এর core skill — both directions cultivate। Civitai-তে community-curated negative embeddings huge resource।

প্র ০৪ CFG-এর "burned" / oversaturation problem — কেন হয় এবং কীভাবে fix করা যায়? Imagen-এর dynamic thresholding, SD3-এর rescale, CFG++ — comparison।

CFG-এর dark side — high $w$-এ image distribution থেকে "exit" করে। Production diffusion-এ regular issue।

Why oversaturation occurs:

$\hat\epsilon = \epsilon_{uncond} + w(\epsilon_{cond} - \epsilon_{uncond})$।
$w > 1$ → magnitude $\|\hat\epsilon\| > \|\epsilon_{uncond}\|$ typically।
Larger $\hat\epsilon$ → predicted $\hat{\mathbf{x}}_0$ outside training data range।
Pixel values clipped to $[-1, 1]$ — saturation, posterization।
Especially bright/dark areas: latent space pushed toward extremes।

Visual symptoms:

Skin: plastic, glowing।
Sky: pure white/blue, banding।
Colors: cartoon-like saturation।
Highlights: blown out।
Shadows: solid black।
Overall: "AI look" — uncanny valley।

Imagen's dynamic thresholding (Saharia et al. 2022):

Computed $\hat{\mathbf{x}}_0$ at each step from $\hat\epsilon$।
Compute pixel value percentile $p$ (e.g., 99.5%)।
If $p > $ threshold (e.g., 1.0), rescale: $\hat{\mathbf{x}}_0 / s$ where $s = \max(p, 1)$।
Then clip to $[-1, 1]$।
Preserves bright/dark detail while preventing saturation।

Imagen impact:

Allowed $w$ up to 30 without artifacts (vs 7-10 typical)।
Strong text adherence + clean photorealistic output।
Inspired SD3, other modern models।

Stable Diffusion-এর CFG rescale (Lin et al. 2024):

"Common diffusion noise schedules and sample steps are flawed"।
Observation: SD trained with non-zero terminal SNR — inference assumes pure noise at $t=T$।
Mismatch causes brightness bias।
Fix: rescale CFG output to match unconditional std deviation।
$\hat\epsilon_{rescaled} = \hat\epsilon \cdot \frac{\sigma_{cond}}{\sigma_{cfg}}$।
SD3 incorporates this fix natively।

CFG++ (Chung et al. 2024):

Reformulation as Manifold-Constrained Gradient (MCG)।
Project guidance onto data manifold tangent space।
No oversaturation, even at high $w$।
Better diversity preservation।
Drop-in replacement, no retraining।

Other variants:

APG (Adaptive Projected Guidance, 2024): Adaptive normalization based on noise level।
SAG (Self-Attention Guidance): Internal attention as guidance signal — lower $w$ effective।
PAG (Perturbed Attention Guidance): Better quality at low $w$।
SEG (Smoothed Energy Guidance): Differentiable, training-free।

Practical fixes for SD users:

Lower $w$ (3-5) + better prompt — first try।
Enable CFG rescale in WebUI (default in Automatic1111 ≥ 1.8)।
Use modern models (SD3, Flux) with intrinsic fixes।
Apply post-processing: color correction, levels adjustment।
Mix with img2img low-strength to "tone down"।

SD3's approach:

Rectified Flow training — straight noise schedule, less terminal SNR mismatch।
MMDiT architecture — better text-image binding, less $w$ needed।
Built-in CFG rescale।
Result: $w = 3-5$ produces SDXL $w = 7.5$ quality।

Flux's approach:

Distilled-CFG: guidance baked into model itself।
"Guidance scale" parameter affects model differently — not standard CFG।
$w = 3-3.5$ default, very stable।

Trade-offs:

Dynamic thresholding: simple, effective, may slightly soften details।
CFG rescale: fixes brightness, may not fix saturation completely।
CFG++: best quality, slight compute overhead।
Better base model: best long-term solution।

Research frontier:

"Guidance distillation" — bake into model, no inference cost।
Per-region adaptive guidance।
Reward-model-based guidance (RLHF for diffusion)।
Universal guidance frameworks bridging text, image, audio।

মূল উপলব্ধি: CFG burning artifacts SD ecosystem-এর persistent challenge। ২০২২ থেকে progressive fixes — Imagen → CFG rescale → CFG++ → SD3/Flux intrinsic improvements। Practitioners-এর জন্য — modern models আজ much better, but understanding mechanism enables troubleshooting any era SD।

অনুশীলন

Compute cost: CFG-এর জন্য sampling 2× slow। 50-step DDIM-এ U-Net call কয়টি লাগে CFG সহ ও ছাড়া?

CFG ছাড়া: 50 calls (one per step)।

CFG সহ: 100 calls (one cond + one uncond per step)।

Practical: batch করে ৫০ সম্প্রসারিত batch — wallclock 1.6-1.8× slower (memory bandwidth)।
Test guidance scale: Same prompt + seed-এ $w = 1, 3, 7.5, 15$ চালান। Visual ফলাফল লিখুন।
```
for w in [1, 3, 7.5, 15]:
    img = pipe("a red sports car in mountain road",
               guidance_scale=w,
               generator=torch.manual_seed(42)).images[0]
    img.save(f"cfg_{w}.png")
```
- w=1: vague, may not be red
- w=3: red car, soft, natural
- w=7.5: vivid red, sharp, slight saturation
- w=15: cartoon-like, oversaturated, possible artifacts
Negative prompt design: "A Bangladeshi village in monsoon" — উপযুক্ত negative prompt লিখুন।

Suggested: "blurry, low quality, jpeg artifacts, oversaturated, generic Asian, Indian village, dry land, sunny, modern buildings, cartoon, watermark, text, distorted"।

Cultural specificity-এর জন্য specific exclusions critical।