পাঠ ১৫ · ৩৫-এর মধ্যে · মডিউল ২

Home / AI Courses / Computer Vision / DenseNet & EfficientNet

DenseNet ও EfficientNet

DenseNet & EfficientNet — feature reuse and compound scaling

৭ মিনিট পড়া মাঝারি · Intermediate PyTorch কোডসহ

এই পাঠে যা শিখবেন

DenseNet-এর dense connectivity
EfficientNet-এর compound scaling formula
MBConv block (mobile inverted bottleneck)
কোন architecture কখন

১ · DenseNet — feature reuse

Huang et al. (২০১৭, CVPR best paper). Idea — ResNet-এর skip connection-কে extreme-এ নিয়ে যান।

ResNet block: $x_l = F(x_{l-1}) + x_{l-1}$।
DenseNet block: $x_l = F([x_0, x_1, ..., x_{l-1}])$ — সব আগের layer-এর concatenation.

প্রতিটি new layer — সব previous output access. Connection count: $\binom{L+1}{2}$ for $L$-layer dense block.

কেন্দ্রীয় ধারণা

ResNet — feature add (lossy). DenseNet — feature concat (lossless). প্রতি layer পুরো history দেখে। Parameter অনেক কম, কিন্তু memory বেশি।

২ · DenseNet structure

Dense block: 6-12 layer concat-connected.
Transition layer: dense block-এর মধ্যে — 1×1 conv (channel reduce) + 2×2 pool.
Growth rate $k$: প্রতিটি layer $k$ channel add. Typical 12-32.
Bottleneck: 1×1 → 3×3 within layer.

৩ · DenseNet variants

DenseNet-121: 8M params. Standard.
DenseNet-169, 201, 264: deeper variants.
ResNet-50 (25M, 76% top-1) vs DenseNet-121 (8M, 75% top-1) — 3x kewer params!

৪ · DenseNet-এর benefit

Parameter efficient: feature reuse — same task fewer parameter.
Gradient flow: direct connection layer-1 থেকে layer-L.
Implicit deep supervision: early layer features classifier-এ direct contribute.
Diversified features: later layer different "context" combine.

৫ · DenseNet-এর সমস্যা

Memory hungry: activation tensor concat — train memory প্রায় 2x.
Slower: concatenation overhead, GPU non-friendly.
Implementation complex: ResNet-এর চেয়ে বেশি bookkeeping.

৬ · EfficientNet — compound scaling

Tan & Le (২০১৯, Google). Question — যখন আমরা CNN scale করি — depth, width, resolution-এর কোনটা বাড়াই?

Previous practice: arbitrary. ResNet — depth. Wide ResNet — width. বেশিরভাগ — increase image size.

EfficientNet insight: তিনটিকেই balance করতে হবে। Compound coefficient $\phi$ দিয়ে:

Depth: $d = \alpha^\phi$।
Width: $w = \beta^\phi$।
Resolution: $r = \gamma^\phi$।
Constraint: $\alpha \cdot \beta^2 \cdot \gamma^2 \approx 2$ (FLOPs ~$2^\phi$)।

৭ · EfficientNet B0 → B7

Variant	Resolution	Params	Top-1
B0	224×224	5.3M	77.1%
B1	240	7.8M	79.1%
B3	300	12M	81.6%
B5	456	30M	83.6%
B7	600	66M	84.3%

৮ · MBConv block

EfficientNet-এর core block — Mobile Inverted Bottleneck Convolution (MobileNetV2 origin).

x → 1×1 conv (expand) → BN → SiLU
  → 3×3 depthwise conv → BN → SiLU
  → SE module (channel attention)
  → 1×1 conv (project)  → BN
  → Skip connection

Inverted bottleneck: ResNet-এ ছোট মাঝখানে; এতে — বড় মাঝখানে।
Depthwise sep conv: per-channel spatial → 1×1 cross-channel.
SiLU/Swish: $x \cdot \sigma(x)$ activation.
SE module: Squeeze-and-Excitation — channel-wise attention.

DenseNet-এর "shared library" model — যে কেউ যে কোনো previous chapter-এ যেতে পারে। EfficientNet-এর "balanced diet" — depth, width, resolution-এর সঠিক recipe.

DenseNet — feature reuse via dense concat. EfficientNet — depth/width/resolution-এর coordinated scaling.

৯ · PyTorch-এ EfficientNet

Python · PyTorch

import torch
from torchvision.models import efficientnet_b0, EfficientNet_B0_Weights
from torchvision.models import densenet121, DenseNet121_Weights

# EfficientNet-B0
m = efficientnet_b0(weights=EfficientNet_B0_Weights.IMAGENET1K_V1)
m.eval()
x = torch.randn(1, 3, 224, 224)
with torch.no_grad():
    y = m(x)
print("EfficientNet-B0 out:", y.shape)
print(f"Params: {sum(p.numel() for p in m.parameters()):,}")

# DenseNet-121
m2 = densenet121(weights=DenseNet121_Weights.IMAGENET1K_V1)
m2.eval()
print(f"DenseNet-121 params: {sum(p.numel() for p in m2.parameters()):,}")

EfficientNet-B0 ~5M params — ResNet-50 (25M)-এর 5x ছোট, similar accuracy. Mobile-friendly.

১০ · কোনটা কখন

ResNet-50: general default, well-supported.
DenseNet-121: param-constrained transfer learning, medical imaging.
EfficientNet-B0: mobile, edge — best accuracy/param ratio.
EfficientNet-B5/B7: server-side max accuracy.
EfficientNetV2 (২০২১): faster training, slightly different scaling.

DenseNet memory-heavy — train batch ছোট রাখুন। EfficientNet-এর different B-variant-এ different normalization stats — wrong stats inference accuracy disaster.

ভাবনার প্রশ্ন

প্র ০১ DenseNet "feature reuse" দাবি করে। কিন্তু concat-এর memory cost ResNet-এর চেয়ে অনেক বেশি। কেন তাও পরিগণিত efficient?

এটি apparent paradox — parameter efficient হলেও memory inefficient.

Parameter efficiency:

DenseNet-121: 8M params.
ResNet-50: 25M params.
3x reduction — same accuracy.

কেন param efficient:

Each layer growth rate $k$ = 32 channel add. ResNet block 64-256.
Cumulative concat — feature reuse, no redundant relearning.
Direct supervision early layers — efficient feature use.

Memory cost:

Concat — all activation maps preserve.
Forward pass — concat tensor grow gradually.
Backward pass — all stored.
Memory grows quadratically with depth.

Empirical:

DenseNet-121 train memory ~12 GB (batch 32, 224 input).
ResNet-50 same setting ~6 GB.
2x memory cost despite 3x fewer params.

Memory optimization tricks:

Shared concat memory: recompute concat on backward — trade compute for memory.
Memory-efficient implementation (Pleiss et al., 2017): 4-5x memory reduction.
Gradient checkpointing: standard technique.

Inference vs training:

Inference — no backward, memory acceptable.
Training — main bottleneck.
Mobile inference — DenseNet OK after pruning.

Why param efficiency matters:

Disk storage, model download — params.
Edge device — flash memory limited.
Quantization-friendly — fewer weights.

Why memory matters:

Training cost — GPU rental.
Batch size — too small → noisy gradient.
Multi-GPU complexity.

মূল উপলব্ধি: "Efficient" depends on metric. Param ≠ memory ≠ compute ≠ latency. Engineer-এর কাজ — task-relevant axis-এ optimize.

প্র ০২ EfficientNet-এর compound scaling formula কোথা থেকে এসেছে — empirical heuristic নাকি theoretical?

মোস্টলি empirical, কিন্তু constraint mathematical.

Search procedure (paper-এর Section 3):

Baseline EfficientNet-B0 architecture — Neural Architecture Search-এ obtain.
Fix $\phi = 1$।
Grid search over $\alpha, \beta, \gamma$।
Constraint $\alpha \cdot \beta^2 \cdot \gamma^2 \approx 2$ (FLOPs ~$2^\phi$)।
Best: $\alpha=1.2, \beta=1.1, \gamma=1.15$।

FLOPs constraint origin:

FLOPs ∝ depth × width² × resolution²।
Doubling each: depth ×2 → 2x FLOPs. Width ×2 → 4x. Resolution ×2 → 4x.
$\alpha \cdot \beta^2 \cdot \gamma^2 = 2$ → φ increment = 2x FLOPs.

Why specific values?

$\alpha=1.2$: depth most beneficial gain per FLOP.
$\beta=1.1$: width modest scaling.
$\gamma=1.15$: resolution important কিন্তু compute heavy.
Hardware-dependent — GPU memory access pattern.

Theoretical limitation:

FLOPs counter — actual latency-এর approximate.
Hardware-specific scaling vary.
Memory bandwidth bottleneck capture করে না।

Empirical alternatives:

RegNet (Radosavovic et al., 2020) — different scaling rule.
NFNet — normalization-free, different scaling.
EfficientNetV2 — fused MBConv early, slightly different.

Practical takeaway:

"Don't scale only depth" — VGG era mistake.
Balance > brute force.
Specific α, β, γ task/architecture-specific.

Bangladesh deployment:

Mobile — B0/B1, low resolution OK.
Server — B3/B5.
B7 expensive — only when accuracy critical.

মূল উপলব্ধি: Compound scaling — discipline of "scale everything together". Specific numbers empirical, principle universal.

প্র ০৩ MBConv-এ "inverted bottleneck" (expand → process → project) — ResNet-এর reverse. কেন এটি mobile-friendly?

MobileNetV2 (Sandler et al., 2018)-এর key contribution. Counter-intuitive কিন্তু effective.

ResNet bottleneck:

Wide (256) → narrow (64) → process → wide (256).
Skip connection on wide tensor.
Memory storage — wide tensors.

MobileNet inverted bottleneck:

Narrow (16) → wide (96 = 6×16) → process → narrow (16).
Skip connection on narrow tensor.
Memory storage — narrow tensors.

Why mobile-friendly:

Memory efficiency: only narrow activations stored cross-block. Mobile RAM-limited.
Depthwise spatial: wide expansion-এ depthwise conv (per-channel) — cheap.
Linear bottleneck: last 1×1 — no ReLU. Avoid information collapse in narrow space.

"Linear bottleneck" theory:

Narrow space-এ ReLU — many neuron 0.
Information collapse → manifold corrupt.
Linear projection preserve manifold.
Wide intermediate-এ ReLU OK — redundancy থাকে।

Expansion ratio:

Typical 6× — 16 → 96 → 16.
Smaller (3-4×) — less compute but accuracy drop.
Larger (8-10×) — diminishing return.

SE module addition (EfficientNet):

Squeeze: GAP → 1×1 conv reduce.
Excitation: 1×1 conv → sigmoid scale.
Channel attention — useful channel boost.
~1% accuracy gain, minimal compute.

Activation choice:

MobileNetV2: ReLU6 (clip to 6).
EfficientNet: SiLU/Swish — $x \cdot \sigma(x)$।
Smooth — better gradient.

Hardware optimizations:

Depthwise conv — Apple Neural Engine, Qualcomm Hexagon optimized.
Inverted bottleneck — fewer memory transfers.
Compiler-level fusion — depthwise + pointwise fused kernel.

মূল উপলব্ধি: Mobile architecture — different constraint, different shape. Inverted bottleneck — counter-intuitive insight from optimization-aware design.

প্র ০৪ EfficientNet ImageNet-এ champion ছিল ২০১৯-এ। ২০২৬-এ — ConvNeXt, MaxViT, EVA — কেমন আছে EfficientNet-এর position?

Honest assessment — EfficientNet still relevant কিন্তু leadership বদলে গেছে।

EfficientNet (2019):

B7: 84.3% top-1, 66M params.
Mobile-friendly variants (Lite).
Production-deployed massively.

EfficientNetV2 (2021):

Fused MBConv early stages — faster training.
Progressive resizing — gradual resolution increase during training.
87.3% top-1 (V2-XL).

ConvNeXt (2022, Liu et al.):

"Modernized" ResNet — transformer-inspired tweaks.
Depthwise 7×7, LayerNorm, GELU, fewer normalization.
87.8% top-1 (ConvNeXt-XL).
Pure CNN, transformer-competitive.

MaxViT (2022):

CNN + Transformer hybrid.
Multi-axis attention.
88.5% top-1.

EVA, DINOv2 (2023):

Self-supervised foundation models.
Linear probe 89%+.
Transfer to anything without fine-tune.

EfficientNet-এর current position:

Mobile: still strong — EfficientNet-Lite series.
Edge devices: NPU optimized for MBConv.
Production simplicity: well-tested, documented.
Transfer learning: ResNet-50/EfficientNet-B0 — most common starting point.

Where EfficientNet is being supplanted:

Server-side classification — ConvNeXt, MaxViT.
Object detection backbone — Swin, ConvNeXt.
Foundation model finetune — DINOv2.
Multimodal — CLIP-derived ViT.

Bangladesh-এ practical:

Mobile/Edge — EfficientNet-B0 still best.
Server — ConvNeXt-Small competitive, similar speed.
Research — DINOv2 features worth experimenting.

2-year forecast:

EfficientNet — niche but enduring (mobile).
ConvNeXt-style — "modern CNN" default.
Foundation models — research frontier.
Mamba/RWKV-vision — disruptor potential.

মূল উপলব্ধি: ML architecture aging cycle — 3-5 year leadership. EfficientNet — graceful aging, still useful for specific contexts. Engineer-এর task: track + adapt.

অনুশীলন

Compound calc: EfficientNet B0 → B3 — depth, width, resolution কতটা বাড়ে?

$\phi=3$, so $d=1.2^3=1.73$, $w=1.1^3=1.33$, $r=1.15^3=1.52$। Resolution: 224 × 1.52 ≈ 340 (paper-এ 300).

Pretrained: torchvision থেকে EfficientNet-B0 load করে cat/dog finetune-এর কোড লিখুন।

from torchvision.models import efficientnet_b0
m = efficientnet_b0(weights='DEFAULT')
m.classifier[1] = nn.Linear(1280, 2)  # binary
# train m.classifier only বা full network

ভাবুন: DenseNet-121 medical imaging-এ popular কেন? ResNet-50-এর তুলনায় কী advantage?

Medical dataset ছোট (1K-10K). DenseNet param efficient (8M vs 25M) — overfitting কম। Feature reuse subtle distinction (tumor vs benign) capture-এ ভালো। Stanford CheXNet, etc. DenseNet-121-এ।