পাঠ ৩০ · ৩৫-এর মধ্যে · মডিউল ৪
Home / AI Courses / Computer Vision / ControlNet

ControlNet ও DreamBooth

ControlNet & DreamBooth — controlled and personalized generation
৭ মিনিট পড়া উচ্চ · Advanced diffusers কোড

এই পাঠে যা শিখবেন

  • ControlNet architecture — frozen SD + trainable copy
  • Common conditions — Canny, depth, pose, scribble
  • DreamBooth — subject-driven generation
  • LoRA — lightweight finetune

১ · কেন controlled generation?

Pure text-to-image limitation: prompt-এ "a person doing yoga" — কোন pose? Random. User want specific pose match.

Solution: structural condition — pose skeleton, edge map, depth, scribble. Generation align with input structure.

কেন্দ্রীয় ধারণা

ControlNet = "SD + extra condition channel". Frozen SD weight (preserve generation), trainable copy condition learn. Best of both — pretrained quality + new control.

২ · ControlNet architecture

  • Original SD U-Net frozen.
  • Encoder layers cloned (trainable).
  • Condition (pose/depth/edge) → cloned encoder → adds to original via "zero convolution".
  • Zero conv initialized 0 — start from no influence, gradually learn.
  • Original capability preserved.

৩ · Common conditions

  • Canny edge: input edge map → matching edge generation.
  • Depth: depth map → 3D structure preserve.
  • OpenPose: human skeleton → pose match.
  • Scribble: user sketch → realistic image.
  • Segmentation: mask → fill regions.
  • Normal map: surface orientation.
  • HED, MLSD: alternative edge.

৪ · ControlNet use

Python · diffusers 
from diffusers import StableDiffusionControlNetPipeline, ControlNetModel
import torch
from PIL import Image
import cv2
import numpy as np

# Canny ControlNet
controlnet = ControlNetModel.from_pretrained(
    "lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16
)
pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    controlnet=controlnet,
    torch_dtype=torch.float16
).to("cuda")

# Generate Canny edge from image
img = np.array(Image.open("input.jpg"))
canny = cv2.Canny(img, 100, 200)
canny_img = Image.fromarray(np.stack([canny]*3, axis=-1))

# Generate matching image
result = pipe(
    "A cyberpunk city, neon lights",
    image=canny_img,
    num_inference_steps=30,
).images[0]
result.save("output.png")

    


    

      Input edge map preserve হয়, content text-prompt-অনুযায়ী generate. Composition control + creativity.
    


    
৫ · DreamBooth (২০২২, Google)

    

      "Personalize" generation — your specific subject in any context.
    

    

      Process:
    

    
    
      
  1. 3-5 image of subject (e.g., your pet).
    2. 
      
  2. Unique identifier token: "sks dog".
    3. 
      
  3. Finetune SD — subject + identifier association.
    4. 
      
  4. Generation: "sks dog on Mars" — your specific dog on Mars.
    5. 
    


    
৬ · DreamBooth process

    
    
      
  • Caption: "a photo of sks [class]".
    • 
      
  • Training: 500-1000 step on 3-5 image.
    • 
      
  • Class-prior preservation: regular [class] image co-train — overfit prevent.
    • 
      
  • Cost: ~30 minute single GPU.
    • 
    


    
৭ · LoRA — efficient alternative

    

      Full DreamBooth finetune — entire model. LoRA — only small additions.
    

    
    
      
  • $W' = W + AB$ where rank $r << d$।
    • 
      
  • Original $W$ frozen.
    • 
      
  • Only $A, B$ train — small.
    • 
      
  • LoRA file 5-100 MB (vs full SD 4GB).
    • 
      
  • Easy combine multiple LoRAs (style + subject).
    • 
    


    

      ControlNet = "set the pose/composition rules". DreamBooth = "introduce a specific character". LoRA = "lightweight stylize/personalize". সব সঙ্গে — full creative control.
    


    
    

      
        SD + ControlNet + DreamBooth + LoRA

        
        
        🎨 ControlNet
        structural condition
        
        Canny / depth / pose / scribble
        structure preserve, content generate

        
        
        👤 DreamBooth
        subject personalize
        
        3-5 subject image
        "sks dog" identifier

        
        
        ⚡ LoRA
        lightweight finetune
        
        5-100 MB file
        style + subject combine

        
        
        Combined: "my dog (DreamBooth) doing yoga pose (ControlNet) in jamdani style (LoRA)"
        Full creative control with pretrained foundation
        Each module independent, composable
      
      
SD ecosystem — ControlNet structural, DreamBooth subject, LoRA style. Combinable for full control.

    


    
৮ · Use cases

    
    
      
  • ControlNet:
        
      
          
    • Architecture viz — sketch → photo.
      • 
          
    • Fashion — pose model.
      • 
          
    • Product photography.
      • 
          
    • Comic — consistent pose.
      • 
          
    • Game asset — depth-aware texture.
      • 
        
    
      
    • 
      
  • DreamBooth:
        
      
          
    • Personal avatar generation.
      • 
          
    • Brand mascot consistency.
      • 
          
    • Character art series.
      • 
          
    • Pet portrait.
      • 
        
    
      
    • 
      
  • LoRA:
        
      
          
    • Specific artist style.
      • 
          
    • Anime character.
      • 
          
    • Bangladesh cultural element (jamdani, rickshaw art).
      • 
        
    
      
    • 
    


    
৯ · Bangladesh-specific

    
    
      
  • Jamdani-style LoRA — rickshaw painting aesthetic.
    • 
      
  • Pohela Boishakh themed model.
    • 
      
  • Local celebrity DreamBooth (consent essential).
    • 
      
  • Tourism destination ControlNet.
    • 
      
  • Cultural archive — old photo restoration.
    • 
    


    
১০ · Tools ecosystem

    
    
      
  • Automatic1111 (A1111): popular SD UI. ControlNet, LoRA, DreamBooth — all extension.
    • 
      
  • ComfyUI: node-based. Workflow design.
    • 
      
  • Forge: A1111 fork — faster.
    • 
      
  • Diffusers: Python library — programmatic.
    • 
      
  • Kohya_ss: training tool.
    • 
    


    

      Subject person/celebrity-এর consent without DreamBooth — privacy violation. Bangladesh law gradually addressing. Personal creative use OK, public release careful.
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      
        প্র ০১
        ControlNet "zero convolution" — initialized to 0. কেন এই trick?
      
      

        
Zero conv — ControlNet-এর elegant engineering choice.

        
Problem:

        
    
          
  • SD U-Net pretrained, high quality.
    • 
          
  • Adding new branch — random initialization disturb.
    • 
          
  • Initial training degrade SD output.
    • 
        

        
Zero conv solution:

        
    
          
  • Connection layer initialized to weight = 0.
    • 
          
  • Initial output = 0 (no perturb).
    • 
          
  • Original SD output unchanged at start.
    • 
          
  • Gradient flows — gradually learn to add useful signal.
    • 
        

        
Theoretical justification:

        
    
          
  • Identity mapping at start.
    • 
          
  • Smooth optimization landscape.
    • 
          
  • "Don't fix what's not broken".
    • 
        

        
Empirical:

        
    
          
  • Random init — SD quality drop, recovery slow.
    • 
          
  • Zero init — SD quality preserved, gradual ControlNet gain.
    • 
          
  • Faster convergence.
    • 
        

        
Generalization:

        
    
          
  • "Adapter" pattern — frozen base + trainable adapter.
    • 
          
  • NLP — LoRA, prefix tuning.
    • 
          
  • Common in foundation model fine-tune.
    • 
        

        
মূল উপলব্ধি: "Identity at init" — fine-tune trick widely useful. Preserve pretrained, gradually adapt.

      

    


    

      
        প্র ০২
        DreamBooth "class prior preservation" — কী, কেন essential?
      
      

        
Subject overfit-এর protection.

        
Without prior:

        
    
          
  • 3-5 dog image → "sks dog".
    • 
          
  • Side effect: "dog" general also become like sks!
    • 
          
  • "A dog playing" → looks like sks dog (overfit).
    • 
          
  • Model "lose" general dog concept.
    • 
        

        
Class prior preservation:

        
    
          
  • Regular dog image (200) generated by SD pre-finetune.
    • 
          
  • Co-train: "sks dog" + "a dog".
    • 
          
  • Model learns sks specific while keeping dog general.
    • 
        

        
Loss:

        
    
          
  • Subject loss: predict noise on subject image with "sks dog".
    • 
          
  • Class loss: predict noise on regular image with "a dog".
    • 
          
  • Combined — preserve generality.
    • 
        

        
Empirical:

        
    
          
  • Without: sks dog OK but "any dog" prompts produce sks-like.
    • 
          
  • With: clean separation.
    • 
          
  • Critical for production use.
    • 
        

        
Modern alternatives:

        
    
          
  • Textual inversion — token only, no model update.
    • 
          
  • LoRA — small weights, less catastrophic forgetting.
    • 
          
  • Custom Diffusion — multiple subject.
    • 
        

        
মূল উপলব্ধি: Personalization without forgetting generic — fundamental ML challenge. Prior preservation classic technique.

      

    


    

      
        প্র ০৩
        LoRA-এর rank — typically 4-32. Choice impact?
      
      

        
LoRA hyperparameter tuning.

        
Rank meaning:

        
    
          
  • $W' = W + AB$, $A: d \times r, B: r \times d$।
    • 
          
  • Rank $r$ = added expressivity.
    • 
          
  • Original $W$: $d \times d$ (large).
    • 
          
  • LoRA: $2dr$ (smaller).
    • 
        

        
Low rank (r=4):

        
    
          
  • Few parameter — small file (5MB).
    • 
          
  • Fast train, less overfit.
    • 
          
  • Limited expressivity.
    • 
          
  • Subtle style works.
    • 
        

        
Mid rank (r=16, 32):

        
    
          
  • Balanced.
    • 
          
  • Most common choice.
    • 
          
  • 30-100MB file.
    • 
        

        
High rank (r=64, 128):

        
    
          
  • More expressivity.
    • 
          
  • Larger file.
    • 
          
  • Risk overfit.
    • 
          
  • Complex style/character.
    • 
        

        
Empirical:

        
    
          
  • Style LoRA: r=4-16.
    • 
          
  • Character: r=16-32.
    • 
          
  • Concept: r=32-64.
    • 
          
  • Tuned per task.
    • 
        

        
LoRA combine:

        
    
          
  • Multiple LoRAs add: $W' = W + \alpha_1 A_1 B_1 + \alpha_2 A_2 B_2$।
    • 
          
  • $\alpha$: weight (0-1.5 typical).
    • 
          
  • Style LoRA + character LoRA combine.
    • 
        

        
মূল উপলব্ধি: Rank — capacity dial. Right rank task-specific. Low default, high if needed.

      

    


    

      
        প্র ০৪
        Bangladesh fashion designer — own collection-এর photo-shoot AI দিয়ে generate. ControlNet + DreamBooth pipeline ডিজাইন করুন।
      
      

        
Real Bangladesh fashion AI use case.

        
Goal: custom clothing on diverse model in different settings, without expensive photo shoots.

        
Pipeline:

        
    
          
  1. Clothing dataset: 20-50 image of designer's collection on neutral mannequin.
    2. 
          
  2. DreamBooth/LoRA train: "sks jamdani saree" identifier.
    3. 
          
  3. Pose reference: stock photo of model pose (or 3D render).
    4. 
          
  4. OpenPose extract: pose skeleton.
    5. 
          
  5. ControlNet generate: "sks jamdani saree, professional model, sunset, beach" with pose condition.
    6. 
        

        
Multi-LoRA combine:

        
    
          
  • Clothing LoRA (DreamBooth).
    • 
          
  • Style LoRA (Bangladesh aesthetic).
    • 
          
  • Skin tone LoRA (diverse).
    • 
          
  • Setting LoRA (Bangladesh location).
    • 
        

        
Quality assurance:

        
    
          
  • Model generates 50+ variations per design.
    • 
          
  • Human curate — best 5-10.
    • 
          
  • Inpainting refine specific area (face, hand).
    • 
        

        
Cost analysis:

        
    
          
  • Traditional photo shoot: $1000-5000 per session.
    • 
          
  • AI pipeline: $50-200 (cloud GPU) + curation time.
    • 
          
  • 10x to 100x cost reduction.
    • 
          
  • Iteration speed: hours vs week.
    • 
        

        
Ethical considerations:

        
    
          
  • Real model employment impact — don't replace, augment.
    • 
          
  • Diverse representation — important.
    • 
          
  • Disclose AI-generated.
    • 
          
  • Cultural sensitivity (e.g., religious wear).
    • 
        

        
Bangladesh ecosystem:

        
    
          
  • Aarong, Mukti, Yellow brands experimenting.
    • 
          
  • Daraz e-commerce — product visualization.
    • 
          
  • Local design schools — curriculum integrate.
    • 
        

        
Extensions:

        
    
          
  • Virtual try-on — customer face + outfit.
    • 
          
  • Catalog generation — bulk.
    • 
          
  • Marketing video (animate diff).
    • 
        

        
মূল উপলব্ধি: Generative AI — small business democratizer. Bangladesh fashion industry — global reach via AI. Quality + cultural authenticity = competitive advantage.

      

    

  


  

    
 অনুশীলন

    
    
      
  1. 
        ControlNet test: Canny ControlNet দিয়ে input image-এর edge preserve, প্রিয় style apply.
        
    
          
          
    
            
    Code section ৪-এ। Free Colab GPU দিয়ে চালান।
    
          
    
        
    
      
    2. 

      
  2. 
        LoRA train: Kohya_ss UI, 20 image, "anime style" LoRA — 30 minute training.
        
    
          
          
    
            
    Online tutorial follow. Cloud GPU (RunPod) recommended. Free local impossible without 12GB VRAM.
    
          
    
        
    
      
    3. 

      
  3. 
        ভাবুন: ControlNet vs textual prompt — কখন কোনটা?
        
    
          
          
    
            
    Spatial precision — ControlNet (pose, layout). Style/concept — text prompt. Combined ideal.
    
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন · ABCL TECH-এ আপনার পরবর্তী পদক্ষেপ

    
    

      কোড রানার কাজ না করলে? Google Colab use করুন।
    

  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ২৯ · Stable Diffusion