ControlNet ও DreamBooth
এই পাঠে যা শিখবেন
- ControlNet architecture — frozen SD + trainable copy
- Common conditions — Canny, depth, pose, scribble
- DreamBooth — subject-driven generation
- LoRA — lightweight finetune
১ · কেন controlled generation?
Pure text-to-image limitation: prompt-এ "a person doing yoga" — কোন pose? Random। User want specific pose match।
Solution: structural condition — pose skeleton, edge map, depth, scribble। Generation align with input structure।
ControlNet = "SD + extra condition channel"। Frozen SD weight (preserve generation), trainable copy condition learn। Best of both — pretrained quality + new control।
২ · ControlNet architecture
- Original SD U-Net frozen।
- Encoder layers cloned (trainable)।
- Condition (pose/depth/edge) → cloned encoder → adds to original via "zero convolution"।
- Zero conv initialized 0 — start from no influence, gradually learn।
- Original capability preserved।
৩ · Common conditions
- Canny edge: input edge map → matching edge generation।
- Depth: depth map → 3D structure preserve।
- OpenPose: human skeleton → pose match।
- Scribble: user sketch → realistic image।
- Segmentation: mask → fill regions।
- Normal map: surface orientation।
- HED, MLSD: alternative edge।
৪ · ControlNet use
from diffusers import StableDiffusionControlNetPipeline, ControlNetModel
import torch
from PIL import Image
import cv2
import numpy as np
# Canny ControlNet
controlnet = ControlNetModel.from_pretrained(
"lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16
)
pipe = StableDiffusionControlNetPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5",
controlnet=controlnet,
torch_dtype=torch.float16
).to("cuda")
# Generate Canny edge from image
img = np.array(Image.open("input.jpg"))
canny = cv2.Canny(img, 100, 200)
canny_img = Image.fromarray(np.stack([canny]*3, axis=-1))
# Generate matching image
result = pipe(
"A cyberpunk city, neon lights",
image=canny_img,
num_inference_steps=30,
).images[0]
result.save("output.png")৫ · DreamBooth (২০২২, Google)
"Personalize" generation — your specific subject in any context।
Process:
- 3-5 image of subject (e.g., your pet)।
- Unique identifier token: "sks dog"।
- Finetune SD — subject + identifier association।
- Generation: "sks dog on Mars" — your specific dog on Mars।
৬ · DreamBooth process
- Caption: "a photo of sks [class]"।
- Training: 500-1000 step on 3-5 image।
- Class-prior preservation: regular [class] image co-train — overfit prevent।
- Cost: ~30 minute single GPU।
৭ · LoRA — efficient alternative
Full DreamBooth finetune — entire model। LoRA — only small additions।
- $W' = W + AB$ where rank $r << d$।
- Original $W$ frozen।
- Only $A, B$ train — small।
- LoRA file 5-100 MB (vs full SD 4GB)।
- Easy combine multiple LoRAs (style + subject)।
৮ · Use cases
- ControlNet:
- Architecture viz — sketch → photo।
- Fashion — pose model।
- Product photography।
- Comic — consistent pose।
- Game asset — depth-aware texture।
- DreamBooth:
- Personal avatar generation।
- Brand mascot consistency।
- Character art series।
- Pet portrait।
- LoRA:
- Specific artist style।
- Anime character।
- Bangladesh cultural element (jamdani, rickshaw art)।
৯ · Bangladesh-specific
- Jamdani-style LoRA — rickshaw painting aesthetic।
- Pohela Boishakh themed model।
- Local celebrity DreamBooth (consent essential)।
- Tourism destination ControlNet।
- Cultural archive — old photo restoration।
১০ · Tools ecosystem
- Automatic1111 (A1111): popular SD UI। ControlNet, LoRA, DreamBooth — all extension।
- ComfyUI: node-based. Workflow design।
- Forge: A1111 fork — faster।
- Diffusers: Python library — programmatic।
- Kohya_ss: training tool।
ভাবনার প্রশ্ন
প্র ০১ ControlNet "zero convolution" — initialized to 0। কেন এই trick?
Zero conv — ControlNet-এর elegant engineering choice।
Problem:
- SD U-Net pretrained, high quality।
- Adding new branch — random initialization disturb।
- Initial training degrade SD output।
Zero conv solution:
- Connection layer initialized to weight = 0।
- Initial output = 0 (no perturb)।
- Original SD output unchanged at start।
- Gradient flows — gradually learn to add useful signal।
Theoretical justification:
- Identity mapping at start।
- Smooth optimization landscape।
- "Don't fix what's not broken"।
Empirical:
- Random init — SD quality drop, recovery slow।
- Zero init — SD quality preserved, gradual ControlNet gain।
- Faster convergence।
Generalization:
- "Adapter" pattern — frozen base + trainable adapter।
- NLP — LoRA, prefix tuning।
- Common in foundation model fine-tune।
মূল উপলব্ধি: "Identity at init" — fine-tune trick widely useful। Preserve pretrained, gradually adapt।
প্র ০২ DreamBooth "class prior preservation" — কী, কেন essential?
Subject overfit-এর protection।
Without prior:
- 3-5 dog image → "sks dog"।
- Side effect: "dog" general also become like sks!
- "A dog playing" → looks like sks dog (overfit)।
- Model "lose" general dog concept।
Class prior preservation:
- Regular dog image (200) generated by SD pre-finetune।
- Co-train: "sks dog" + "a dog"।
- Model learns sks specific while keeping dog general।
Loss:
- Subject loss: predict noise on subject image with "sks dog"।
- Class loss: predict noise on regular image with "a dog"।
- Combined — preserve generality।
Empirical:
- Without: sks dog OK but "any dog" prompts produce sks-like।
- With: clean separation।
- Critical for production use।
Modern alternatives:
- Textual inversion — token only, no model update।
- LoRA — small weights, less catastrophic forgetting।
- Custom Diffusion — multiple subject।
মূল উপলব্ধি: Personalization without forgetting generic — fundamental ML challenge। Prior preservation classic technique।
প্র ০৩ LoRA-এর rank — typically 4-32। Choice impact?
LoRA hyperparameter tuning।
Rank meaning:
- $W' = W + AB$, $A: d \times r, B: r \times d$।
- Rank $r$ = added expressivity।
- Original $W$: $d \times d$ (large)।
- LoRA: $2dr$ (smaller)।
Low rank (r=4):
- Few parameter — small file (5MB)।
- Fast train, less overfit।
- Limited expressivity।
- Subtle style works।
Mid rank (r=16, 32):
- Balanced।
- Most common choice।
- 30-100MB file।
High rank (r=64, 128):
- More expressivity।
- Larger file।
- Risk overfit।
- Complex style/character।
Empirical:
- Style LoRA: r=4-16।
- Character: r=16-32।
- Concept: r=32-64।
- Tuned per task।
LoRA combine:
- Multiple LoRAs add: $W' = W + \alpha_1 A_1 B_1 + \alpha_2 A_2 B_2$।
- $\alpha$: weight (0-1.5 typical)।
- Style LoRA + character LoRA combine।
মূল উপলব্ধি: Rank — capacity dial। Right rank task-specific। Low default, high if needed।
প্র ০৪ Bangladesh fashion designer — own collection-এর photo-shoot AI দিয়ে generate। ControlNet + DreamBooth pipeline ডিজাইন করুন।
Real Bangladesh fashion AI use case।
Goal: custom clothing on diverse model in different settings, without expensive photo shoots।
Pipeline:
- Clothing dataset: 20-50 image of designer's collection on neutral mannequin।
- DreamBooth/LoRA train: "sks jamdani saree" identifier।
- Pose reference: stock photo of model pose (or 3D render)।
- OpenPose extract: pose skeleton।
- ControlNet generate: "sks jamdani saree, professional model, sunset, beach" with pose condition।
Multi-LoRA combine:
- Clothing LoRA (DreamBooth)।
- Style LoRA (Bangladesh aesthetic)।
- Skin tone LoRA (diverse)।
- Setting LoRA (Bangladesh location)।
Quality assurance:
- Model generates 50+ variations per design।
- Human curate — best 5-10।
- Inpainting refine specific area (face, hand)।
Cost analysis:
- Traditional photo shoot: $1000-5000 per session।
- AI pipeline: $50-200 (cloud GPU) + curation time।
- 10x to 100x cost reduction।
- Iteration speed: hours vs week।
Ethical considerations:
- Real model employment impact — don't replace, augment।
- Diverse representation — important।
- Disclose AI-generated.
- Cultural sensitivity (e.g., religious wear)।
Bangladesh ecosystem:
- Aarong, Mukti, Yellow brands experimenting।
- Daraz e-commerce — product visualization।
- Local design schools — curriculum integrate।
Extensions:
- Virtual try-on — customer face + outfit।
- Catalog generation — bulk।
- Marketing video (animate diff)।
মূল উপলব্ধি: Generative AI — small business democratizer। Bangladesh fashion industry — global reach via AI। Quality + cultural authenticity = competitive advantage।
অনুশীলন
-
ControlNet test: Canny ControlNet দিয়ে input image-এর edge preserve, প্রিয় style apply।
Code section ৪-এ। Free Colab GPU দিয়ে চালান।
-
LoRA train: Kohya_ss UI, 20 image, "anime style" LoRA — 30 minute training।
Online tutorial follow। Cloud GPU (RunPod) recommended। Free local impossible without 12GB VRAM।
-
ভাবুন: ControlNet vs textual prompt — কখন কোনটা?
Spatial precision — ControlNet (pose, layout)। Style/concept — text prompt। Combined ideal।
আরও পড়ুন · ABCL TECH-এ আপনার পরবর্তী পদক্ষেপ
- পাঠ ৩১ · Face recognition পরবর্তী পাঠCV-র classic application।
- পাঠ ২৯ · Stable Diffusion আগের পাঠFoundation।
- সব AI Courses দেখুন ABCL TECHসব কোর্স।