পাঠ ১৫ · ৩৫-এর মধ্যে · মডিউল ২
Home / AI Courses / Computer Vision / DenseNet & EfficientNet

DenseNet ও EfficientNet

DenseNet & EfficientNet — feature reuse and compound scaling
৭ মিনিট পড়া মাঝারি · Intermediate PyTorch কোডসহ

এই পাঠে যা শিখবেন

  • DenseNet-এর dense connectivity
  • EfficientNet-এর compound scaling formula
  • MBConv block (mobile inverted bottleneck)
  • কোন architecture কখন

১ · DenseNet — feature reuse

Huang et al. (২০১৭, CVPR best paper)। Idea — ResNet-এর skip connection-কে extreme-এ নিয়ে যান।

  • ResNet block: $x_l = F(x_{l-1}) + x_{l-1}$।
  • DenseNet block: $x_l = F([x_0, x_1, ..., x_{l-1}])$ — সব আগের layer-এর concatenation।

প্রতিটি new layer — সব previous output access। Connection count: $\binom{L+1}{2}$ for $L$-layer dense block।

কেন্দ্রীয় ধারণা

ResNet — feature add (lossy)। DenseNet — feature concat (lossless)। প্রতি layer পুরো history দেখে। Parameter অনেক কম, কিন্তু memory বেশি।

২ · DenseNet structure

  • Dense block: 6-12 layer concat-connected।
  • Transition layer: dense block-এর মধ্যে — 1×1 conv (channel reduce) + 2×2 pool।
  • Growth rate $k$: প্রতিটি layer $k$ channel add। Typical 12-32।
  • Bottleneck: 1×1 → 3×3 within layer।

৩ · DenseNet variants

  • DenseNet-121: 8M params। Standard।
  • DenseNet-169, 201, 264: deeper variants।
  • ResNet-50 (25M, 76% top-1) vs DenseNet-121 (8M, 75% top-1) — 3x kewer params!

৪ · DenseNet-এর benefit

  • Parameter efficient: feature reuse — same task fewer parameter।
  • Gradient flow: direct connection layer-1 থেকে layer-L।
  • Implicit deep supervision: early layer features classifier-এ direct contribute।
  • Diversified features: later layer different "context" combine।

৫ · DenseNet-এর সমস্যা

  • Memory hungry: activation tensor concat — train memory প্রায় 2x।
  • Slower: concatenation overhead, GPU non-friendly।
  • Implementation complex: ResNet-এর চেয়ে বেশি bookkeeping।

৬ · EfficientNet — compound scaling

Tan & Le (২০১৯, Google)। Question — যখন আমরা CNN scale করি — depth, width, resolution-এর কোনটা বাড়াই?

Previous practice: arbitrary। ResNet — depth। Wide ResNet — width। বেশিরভাগ — increase image size।

EfficientNet insight: তিনটিকেই balance করতে হবে। Compound coefficient $\phi$ দিয়ে:

  • Depth: $d = \alpha^\phi$।
  • Width: $w = \beta^\phi$।
  • Resolution: $r = \gamma^\phi$।
  • Constraint: $\alpha \cdot \beta^2 \cdot \gamma^2 \approx 2$ (FLOPs ~$2^\phi$)।

৭ · EfficientNet B0 → B7

VariantResolutionParamsTop-1
B0224×2245.3M77.1%
B12407.8M79.1%
B330012M81.6%
B545630M83.6%
B760066M84.3%

৮ · MBConv block

EfficientNet-এর core block — Mobile Inverted Bottleneck Convolution (MobileNetV2 origin)। 

x → 1×1 conv (expand) → BN → SiLU
  → 3×3 depthwise conv → BN → SiLU
  → SE module (channel attention)
  → 1×1 conv (project)  → BN
  → Skip connection
  • Inverted bottleneck: ResNet-এ ছোট মাঝখানে; এতে — বড় মাঝখানে।
  • Depthwise sep conv: per-channel spatial → 1×1 cross-channel।
  • SiLU/Swish: $x \cdot \sigma(x)$ activation।
  • SE module: Squeeze-and-Excitation — channel-wise attention।
DenseNet-এর "shared library" model — যে কেউ যে কোনো previous chapter-এ যেতে পারে। EfficientNet-এর "balanced diet" — depth, width, resolution-এর সঠিক recipe।
DenseNet feature concat · EfficientNet compound scaling DenseNet L1 L2 L3 L4 all-to-all concat feature reuse · param efficient EfficientNet compound scaling Three axes scale together Depth: d = α^φ Width: w = β^φ Resolution: r = γ^φ α·β²·γ² ≈ 2 B0–B7: φ ∈ {0..7} DenseNet — feature reuse architecture · EfficientNet — scaling discipline
DenseNet — feature reuse via dense concat। EfficientNet — depth/width/resolution-এর coordinated scaling।

৯ · PyTorch-এ EfficientNet

Python · PyTorch 
import torch
from torchvision.models import efficientnet_b0, EfficientNet_B0_Weights
from torchvision.models import densenet121, DenseNet121_Weights

# EfficientNet-B0
m = efficientnet_b0(weights=EfficientNet_B0_Weights.IMAGENET1K_V1)
m.eval()
x = torch.randn(1, 3, 224, 224)
with torch.no_grad():
    y = m(x)
print("EfficientNet-B0 out:", y.shape)
print(f"Params: {sum(p.numel() for p in m.parameters()):,}")

# DenseNet-121
m2 = densenet121(weights=DenseNet121_Weights.IMAGENET1K_V1)
m2.eval()
print(f"DenseNet-121 params: {sum(p.numel() for p in m2.parameters()):,}")

    


    

      EfficientNet-B0 ~5M params — ResNet-50 (25M)-এর 5x ছোট, similar accuracy। Mobile-friendly।
    


    
১০ · কোনটা কখন

    
    
      
  • ResNet-50: general default, well-supported।
    • 
      
  • DenseNet-121: param-constrained transfer learning, medical imaging।
    • 
      
  • EfficientNet-B0: mobile, edge — best accuracy/param ratio।
    • 
      
  • EfficientNet-B5/B7: server-side max accuracy।
    • 
      
  • EfficientNetV2 (২০২১): faster training, slightly different scaling।
    • 
    


    

      DenseNet memory-heavy — train batch ছোট রাখুন। EfficientNet-এর different B-variant-এ different normalization stats — wrong stats inference accuracy disaster।
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      
        প্র ০১
        DenseNet "feature reuse" দাবি করে। কিন্তু concat-এর memory cost ResNet-এর চেয়ে অনেক বেশি। কেন তাও পরিগণিত efficient?
      
      

        

          এটি apparent paradox — parameter efficient হলেও memory inefficient।
        

        

          Parameter efficiency:
        

        
    
          
  • DenseNet-121: 8M params।
    • 
          
  • ResNet-50: 25M params।
    • 
          
  • 3x reduction — same accuracy।
    • 
        

        

          কেন param efficient:
        

        
    
          
  • Each layer growth rate $k$ = 32 channel add। ResNet block 64-256।
    • 
          
  • Cumulative concat — feature reuse, no redundant relearning।
    • 
          
  • Direct supervision early layers — efficient feature use।
    • 
        

        

          Memory cost:
        

        
    
          
  • Concat — all activation maps preserve।
    • 
          
  • Forward pass — concat tensor grow gradually।
    • 
          
  • Backward pass — all stored।
    • 
          
  • Memory grows quadratically with depth।
    • 
        

        

          Empirical:
        

        
    
          
  • DenseNet-121 train memory ~12 GB (batch 32, 224 input)।
    • 
          
  • ResNet-50 same setting ~6 GB।
    • 
          
  • 2x memory cost despite 3x fewer params।
    • 
        

        

          Memory optimization tricks:
        

        
    
          
  • Shared concat memory: recompute concat on backward — trade compute for memory।
    • 
          
  • Memory-efficient implementation (Pleiss et al., 2017): 4-5x memory reduction।
    • 
          
  • Gradient checkpointing: standard technique।
    • 
        

        

          Inference vs training:
        

        
    
          
  • Inference — no backward, memory acceptable।
    • 
          
  • Training — main bottleneck।
    • 
          
  • Mobile inference — DenseNet OK after pruning।
    • 
        

        

          Why param efficiency matters:
        

        
    
          
  • Disk storage, model download — params।
    • 
          
  • Edge device — flash memory limited।
    • 
          
  • Quantization-friendly — fewer weights।
    • 
        

        

          Why memory matters:
        

        
    
          
  • Training cost — GPU rental।
    • 
          
  • Batch size — too small → noisy gradient।
    • 
          
  • Multi-GPU complexity।
    • 
        

        

          মূল উপলব্ধি: "Efficient" depends on metric। Param ≠ memory ≠ compute ≠ latency। Engineer-এর কাজ — task-relevant axis-এ optimize।
        

      

    


    

      
        প্র ০২
        EfficientNet-এর compound scaling formula কোথা থেকে এসেছে — empirical heuristic নাকি theoretical?
      
      

        

          মোস্টলি empirical, কিন্তু constraint mathematical।
        

        

          Search procedure (paper-এর Section 3):
        

        
    
          
  1. Baseline EfficientNet-B0 architecture — Neural Architecture Search-এ obtain।
    2. 
          
  2. Fix $\phi = 1$।
    3. 
          
  3. Grid search over $\alpha, \beta, \gamma$।
    4. 
          
  4. Constraint $\alpha \cdot \beta^2 \cdot \gamma^2 \approx 2$ (FLOPs ~$2^\phi$)।
    5. 
          
  5. Best: $\alpha=1.2, \beta=1.1, \gamma=1.15$।
    6. 
        

        

          FLOPs constraint origin:
        

        
    
          
  • FLOPs ∝ depth × width² × resolution²।
    • 
          
  • Doubling each: depth ×2 → 2x FLOPs। Width ×2 → 4x। Resolution ×2 → 4x।
    • 
          
  • $\alpha \cdot \beta^2 \cdot \gamma^2 = 2$ → φ increment = 2x FLOPs।
    • 
        

        

          Why specific values?
        

        
    
          
  • $\alpha=1.2$: depth most beneficial gain per FLOP।
    • 
          
  • $\beta=1.1$: width modest scaling।
    • 
          
  • $\gamma=1.15$: resolution important কিন্তু compute heavy।
    • 
          
  • Hardware-dependent — GPU memory access pattern।
    • 
        

        

          Theoretical limitation:
        

        
    
          
  • FLOPs counter — actual latency-এর approximate।
    • 
          
  • Hardware-specific scaling vary।
    • 
          
  • Memory bandwidth bottleneck capture করে না।
    • 
        

        

          Empirical alternatives:
        

        
    
          
  • RegNet (Radosavovic et al., 2020) — different scaling rule।
    • 
          
  • NFNet — normalization-free, different scaling।
    • 
          
  • EfficientNetV2 — fused MBConv early, slightly different।
    • 
        

        

          Practical takeaway:
        

        
    
          
  • "Don't scale only depth" — VGG era mistake।
    • 
          
  • Balance > brute force।
    • 
          
  • Specific α, β, γ task/architecture-specific।
    • 
        

        

          Bangladesh deployment:
        

        
    
          
  • Mobile — B0/B1, low resolution OK।
    • 
          
  • Server — B3/B5।
    • 
          
  • B7 expensive — only when accuracy critical।
    • 
        

        

          মূল উপলব্ধি: Compound scaling — discipline of "scale everything together"। Specific numbers empirical, principle universal।
        

      

    


    

      
        প্র ০৩
        MBConv-এ "inverted bottleneck" (expand → process → project) — ResNet-এর reverse। কেন এটি mobile-friendly?
      
      

        

          MobileNetV2 (Sandler et al., 2018)-এর key contribution। Counter-intuitive কিন্তু effective।
        

        

          ResNet bottleneck:
        

        
    
          
  • Wide (256) → narrow (64) → process → wide (256)।
    • 
          
  • Skip connection on wide tensor।
    • 
          
  • Memory storage — wide tensors।
    • 
        

        

          MobileNet inverted bottleneck:
        

        
    
          
  • Narrow (16) → wide (96 = 6×16) → process → narrow (16)।
    • 
          
  • Skip connection on narrow tensor।
    • 
          
  • Memory storage — narrow tensors।
    • 
        

        

          Why mobile-friendly:
        

        
    
          
  • Memory efficiency: only narrow activations stored cross-block। Mobile RAM-limited।
    • 
          
  • Depthwise spatial: wide expansion-এ depthwise conv (per-channel) — cheap।
    • 
          
  • Linear bottleneck: last 1×1 — no ReLU। Avoid information collapse in narrow space।
    • 
        

        

          "Linear bottleneck" theory:
        

        
    
          
  • Narrow space-এ ReLU — many neuron 0।
    • 
          
  • Information collapse → manifold corrupt।
    • 
          
  • Linear projection preserve manifold।
    • 
          
  • Wide intermediate-এ ReLU OK — redundancy থাকে।
    • 
        

        

          Expansion ratio:
        

        
    
          
  • Typical 6× — 16 → 96 → 16।
    • 
          
  • Smaller (3-4×) — less compute but accuracy drop।
    • 
          
  • Larger (8-10×) — diminishing return।
    • 
        

        

          SE module addition (EfficientNet):
        

        
    
          
  • Squeeze: GAP → 1×1 conv reduce।
    • 
          
  • Excitation: 1×1 conv → sigmoid scale।
    • 
          
  • Channel attention — useful channel boost।
    • 
          
  • ~1% accuracy gain, minimal compute।
    • 
        

        

          Activation choice:
        

        
    
          
  • MobileNetV2: ReLU6 (clip to 6)।
    • 
          
  • EfficientNet: SiLU/Swish — $x \cdot \sigma(x)$।
    • 
          
  • Smooth — better gradient।
    • 
        

        

          Hardware optimizations:
        

        
    
          
  • Depthwise conv — Apple Neural Engine, Qualcomm Hexagon optimized।
    • 
          
  • Inverted bottleneck — fewer memory transfers।
    • 
          
  • Compiler-level fusion — depthwise + pointwise fused kernel।
    • 
        

        

          মূল উপলব্ধি: Mobile architecture — different constraint, different shape। Inverted bottleneck — counter-intuitive insight from optimization-aware design।
        

      

    


    

      
        প্র ০৪
        EfficientNet ImageNet-এ champion ছিল ২০১৯-এ। ২০২৬-এ — ConvNeXt, MaxViT, EVA — কেমন আছে EfficientNet-এর position?
      
      

        

          Honest assessment — EfficientNet still relevant কিন্তু leadership বদলে গেছে।
        

        

          EfficientNet (2019):
        

        
    
          
  • B7: 84.3% top-1, 66M params।
    • 
          
  • Mobile-friendly variants (Lite)।
    • 
          
  • Production-deployed massively।
    • 
        

        

          EfficientNetV2 (2021):
        

        
    
          
  • Fused MBConv early stages — faster training।
    • 
          
  • Progressive resizing — gradual resolution increase during training।
    • 
          
  • 87.3% top-1 (V2-XL)।
    • 
        

        

          ConvNeXt (2022, Liu et al.):
        

        
    
          
  • "Modernized" ResNet — transformer-inspired tweaks।
    • 
          
  • Depthwise 7×7, LayerNorm, GELU, fewer normalization।
    • 
          
  • 87.8% top-1 (ConvNeXt-XL)।
    • 
          
  • Pure CNN, transformer-competitive।
    • 
        

        

          MaxViT (2022):
        

        
    
          
  • CNN + Transformer hybrid।
    • 
          
  • Multi-axis attention।
    • 
          
  • 88.5% top-1।
    • 
        

        

          EVA, DINOv2 (2023):
        

        
    
          
  • Self-supervised foundation models।
    • 
          
  • Linear probe 89%+।
    • 
          
  • Transfer to anything without fine-tune।
    • 
        

        

          EfficientNet-এর current position:
        

        
    
          
  • Mobile: still strong — EfficientNet-Lite series।
    • 
          
  • Edge devices: NPU optimized for MBConv।
    • 
          
  • Production simplicity: well-tested, documented।
    • 
          
  • Transfer learning: ResNet-50/EfficientNet-B0 — most common starting point।
    • 
        

        

          Where EfficientNet is being supplanted:
        

        
    
          
  • Server-side classification — ConvNeXt, MaxViT।
    • 
          
  • Object detection backbone — Swin, ConvNeXt।
    • 
          
  • Foundation model finetune — DINOv2।
    • 
          
  • Multimodal — CLIP-derived ViT।
    • 
        

        

          Bangladesh-এ practical:
        

        
    
          
  • Mobile/Edge — EfficientNet-B0 still best।
    • 
          
  • Server — ConvNeXt-Small competitive, similar speed।
    • 
          
  • Research — DINOv2 features worth experimenting।
    • 
        

        

          2-year forecast:
        

        
    
          
  • EfficientNet — niche but enduring (mobile)।
    • 
          
  • ConvNeXt-style — "modern CNN" default।
    • 
          
  • Foundation models — research frontier।
    • 
          
  • Mamba/RWKV-vision — disruptor potential।
    • 
        

        

          মূল উপলব্ধি: ML architecture aging cycle — 3-5 year leadership। EfficientNet — graceful aging, still useful for specific contexts। Engineer-এর task: track + adapt।
        

      

    

  


  

    
 অনুশীলন

    
    
      
  1. 
        Compound calc: EfficientNet B0 → B3 — depth, width, resolution কতটা বাড়ে?
        
    
          
          
    
            
    $\phi=3$, so $d=1.2^3=1.73$, $w=1.1^3=1.33$, $r=1.15^3=1.52$। Resolution: 224 × 1.52 ≈ 340 (paper-এ 300)।
    
          
    
        
    
      
    2. 

      
  2. 
        Pretrained: torchvision থেকে EfficientNet-B0 load করে cat/dog finetune-এর কোড লিখুন।
        
    
          
          
    
            
    from torchvision.models import efficientnet_b0
m = efficientnet_b0(weights='DEFAULT')
m.classifier[1] = nn.Linear(1280, 2)  # binary
# train m.classifier only বা full network
    
          
    
        
    
      
    3. 

      
  3. 
        ভাবুন: DenseNet-121 medical imaging-এ popular কেন? ResNet-50-এর তুলনায় কী advantage?
        
    
          
          
    
            
    Medical dataset ছোট (1K-10K)। DenseNet param efficient (8M vs 25M) — overfitting কম। Feature reuse subtle distinction (tumor vs benign) capture-এ ভালো। Stanford CheXNet, etc. DenseNet-121-এ।
    
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন · ABCL TECH-এ আপনার পরবর্তী পদক্ষেপ

    

    

      কোড রানার কাজ না করলে? ব্রাউজারে কাজ না করলে
      Google Colab
      ব্যবহার করুন — Google-এর ফ্রি অনলাইন Python পরিবেশ, শুধু Gmail অ্যাকাউন্ট লাগে।
    

  







  
  
     এই পাঠে ভুল পেয়েছেন? জানান
  





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ১৪ · ResNet