পাঠ ১৪ · ৩৫-এর মধ্যে · মডিউল ২

Home / AI Courses / Computer Vision / ResNet

ResNet — skip connection-এর শক্তি

ResNet — residual learning, the deepest practical CNN

৮ মিনিট পড়া মাঝারি · Intermediate PyTorch কোডসহ

এই পাঠে যা শিখবেন

Residual learning-এর গাণিতিক intuition
Basic block ও bottleneck block-এর architecture
ResNet-18, 34, 50, 101, 152 — কোনটা কখন
PyTorch-এ ResNet implementation

১ · Degradation problem (revisit)

VGG-এর experiment — 19 layer-এর পরে accuracy কমে। Counter-intuitive — capacity বাড়লে কেন accuracy কম?

He et al.-এর observation: "যদি deeper network-এর extra layer identity হত — তাহলে অন্তত equal accuracy থাকা উচিত। Worse মানে — optimization fail."

কেন্দ্রীয় ধারণা

Identity mapping বানানো কঠিন non-linear network-এ। তাই — identity-কে easy default বানিয়ে নিন। "Residual" শেখাতে দিন — পার্থক্যটুকু।

২ · Residual learning

Standard CNN block: input $x$ → $F(x)$ output.

ResNet block:

$$y = F(x) + x$$

$F(x)$ = Conv-BN-ReLU-Conv-BN. "Residual".
$x$ — direct skip connection.
Output $y$ — both summed.

Key insight: network-এর শেখা কাজ এখন $F(x)$ — যা residual. Identity যদি optimal — $F(x) = 0$ শেখা সহজ (just zero out the weights).

৩ · Why does it work?

Gradient flow: backprop-এ skip path-এর gradient direct early layer-এ পৌঁছে। Vanishing gradient problem dissolved.
Smoother loss landscape: Li et al. (২০১৮) visualization — ResNet loss surface অনেক বেশি smooth.
Implicit ensemble: $2^n$ paths through n blocks — multiple sub-networks ensemble.
Easy identity: degradation problem direct address.

৪ · Basic block (ResNet-18, 34)

x → Conv 3×3 → BN → ReLU → Conv 3×3 → BN → (+) → ReLU → output
       └─────────── skip ─────────────────────┘

Two 3×3 conv + skip. ResNet-18 ও 34-এ ব্যবহৃত।

৫ · Bottleneck block (ResNet-50, 101, 152)

x → Conv 1×1 (compress) → BN → ReLU
  → Conv 3×3 (process)  → BN → ReLU
  → Conv 1×1 (expand)   → BN → (+) → ReLU → output
   └─────────────── skip ──────────────┘

Inception-এর bottleneck idea: 1×1 → 3×3 → 1×1. Compute সাশ্রয়। 256 → 64 → 64 → 256 typical.

৬ · ResNet variants

Variant	Block type	Params	Top-1
ResNet-18	Basic	11.7M	69.8%
ResNet-34	Basic	21.8M	73.3%
ResNet-50	Bottleneck	25.6M	76.1%
ResNet-101	Bottleneck	44.5M	77.4%
ResNet-152	Bottleneck	60.2M	78.3%

ResNet = "Don't reinvent identity, just learn what to add to it." একজন editor-এর মতো — original draft preserve, শুধু changes mark. Engineering elegance-এর peak.

ResNet block — residual transformation $F(x)$ + identity skip. Bottleneck-এ 1×1 দিয়ে compute সাশ্রয়।

৭ · PyTorch-এ ResNet block

Python · PyTorch

import torch
import torch.nn as nn

class BasicBlock(nn.Module):
    def __init__(self, ch):
        super().__init__()
        self.conv1 = nn.Conv2d(ch, ch, 3, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(ch)
        self.conv2 = nn.Conv2d(ch, ch, 3, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(ch)
        self.relu = nn.ReLU(inplace=True)
    def forward(self, x):
        identity = x
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out = out + identity                # residual add
        return self.relu(out)

block = BasicBlock(64)
x = torch.randn(1, 64, 56, 56)
y = block(x)
print("In:", x.shape, "Out:", y.shape)

# Pretrained ResNet-50
from torchvision.models import resnet50, ResNet50_Weights
m = resnet50(weights=ResNet50_Weights.IMAGENET1K_V2)
n = sum(p.numel() for p in m.parameters())
print(f"ResNet-50 params: {n:,}")           # ~25.5M

Residual block out + identity — মাত্র এই একটি line CV-কে বদলে দিয়েছে। torchvision-এর ResNet-50 — ImageNet pretrained, transfer learning-এর gold standard.

৮ · ResNet-এর variants

ResNet v2 (২০১৬, He et al.): pre-activation — BN-ReLU আগে, conv পরে। Slightly better.
ResNeXt (২০১৭): grouped convolution. 32-group, similar params, +1% accuracy.
Wide ResNet: deep-এর বদলে wider — 50-layer wider better than 152 thin.
Squeeze-and-Excitation ResNet (SENet): channel attention add. ImageNet 2017 winner.

৯ · Modern uses

Object detection: Faster R-CNN, Mask R-CNN — ResNet-50/101 backbone.
Segmentation: FCN, DeepLab, U-Net++ — ResNet encoder.
Pose estimation: HRNet — ResNet-derived.
Self-supervised: MoCo, SimCLR, BYOL — ResNet-50 backbone.
Medical: chest X-ray, retinal — ResNet-50 transfer learning.
Style transfer: ResNet feature also use.

ResNet AI-র সবচেয়ে cited paper-এর একটি (১৬০K+ citations). ২০২৬-এ এসেও — production CNN-এর majority ResNet backbone. Mastery essential.

ভাবনার প্রশ্ন

প্র ০১ ResNet-এর skip connection identity-add. কিন্তু channel mismatch (64 → 128) হলে কী হয়? Spatial size mismatch হলে?

এটি ResNet implementation-এর crucial detail. Naive skip every-block-এ কাজ করে না।

Channel mismatch:

ResNet-এ stage transition-এ channel double (64 → 128 → 256 → 512).
Direct add impossible — different channel count.
Solution: 1×1 conv on skip path — channel project.

Spatial mismatch:

Stage transition-এ 2x downsample (stride=2).
Skip-ও same downsample চাই।
Solution: 1×1 stride-2 conv on skip path.

Two skip variants:

# Identity skip (when shapes match)
out = F(x) + x

# Projection skip (when shapes mismatch)
out = F(x) + Conv1x1_stride2(x)

Original paper-এ option (A):

Zero-padding for extra channels.
Spatial — every-other element subsample.
No extra parameter.
Slightly worse accuracy.

Option (B): Projection only at downsample (default):

Within-stage skip = identity.
Between-stage skip = 1×1 conv projection.
Standard ResNet implementation.

Option (C): Projection always:

Every block-এ 1×1 projection.
Marginal accuracy gain.
Param overhead তুলনামূলক বেশি।

Code structure:

class ResBlock(nn.Module):
    def __init__(self, in_ch, out_ch, stride=1):
        super().__init__()
        self.conv1 = nn.Conv2d(in_ch, out_ch, 3, stride, 1)
        self.conv2 = nn.Conv2d(out_ch, out_ch, 3, 1, 1)

        # Projection skip if needed
        if stride != 1 or in_ch != out_ch:
            self.skip = nn.Conv2d(in_ch, out_ch, 1, stride)
        else:
            self.skip = nn.Identity()

Pre-activation vs post-activation:

Original: Conv-BN-ReLU-Conv-BN + skip → ReLU.
v2: BN-ReLU-Conv-BN-ReLU-Conv + skip.
v2 — gradient flow better, slightly higher accuracy.

Modern implementations:

torchvision ResNet — option (B).
Timm library — multiple variants.
Custom paper — task-specific tweaks.

মূল উপলব্ধি: "Skip connection trivial" misconception. Implementation-এ subtle, performance impact noticeable. Engineering attention to detail.

প্র ০২ "Implicit ensemble" hypothesis — ResNet-কে $2^n$ paths-এর ensemble বলে দেখা যায়। এই interpretation মাথায় কী implication?

Veit et al. (২০১৬, NeurIPS) — "Residual Networks Behave Like Ensembles". সর্বাধিক influential interpretation.

Path count:

Each ResNet block — দু'টি option: take F(x) ও skip-only.
$n$ blocks → $2^n$ unique paths.
ResNet-152 — $2^{50}+$ paths!

Empirical evidence:

Random block skip — accuracy gentle drop.
VGG block skip — catastrophic drop.
ResNet — robust to layer ablation.

Length distribution of effective paths:

Path length distribution — binomial.
Most paths length n/2 (15-25 for ResNet-50).
Very short ও very long paths rare.
Effective depth অনেক কম than nominal depth.

Implications:

Vanishing gradient: short path-এ direct gradient — primary value.
Stochastic depth: randomly skip block during training (DropPath). Same intuition.
Layer pruning: some block remove without breaking — production-এ smaller model.

Stochastic Depth (Huang et al., 2016):

Training-এ 50% probability skip a block.
Inference-এ — সব block.
Acts as regularization + ensemble.
Train ResNet-1202!

Modern extension:

DropPath: ConvNeXt, Swin-এ standard.
RegNetY: path-level regularization.
NFNet: Normalizer-Free + scaled residual.

Theoretical depth:

Greff et al. — "highway/ResNet performs iterative refinement".
Each block — small refinement to running representation.
Different from "deep abstraction" classical view.

Counter-evidence (Wu et al., 2018):

ResNet-152 reduce করে depth — accuracy drops.
"Pure ensemble" view incomplete.
Block-level interaction matters.

মূল উপলব্ধি: Network-এর behavior often defies architectural intent. ResNet "deep" শোনায় কিন্তু "ensemble" আচরণ। Empirical understanding-ই ground truth.

প্র ০৩ ResNet-152 ImageNet-এ 78%. ResNet-200, 1000 — কেন popular না? Diminishing returns কোথায়?

এটি architecture scaling-এর fundamental observation.

ResNet depth vs accuracy:

ResNet-18: 70%.
ResNet-34: 73% (+3%).
ResNet-50: 76% (+3%).
ResNet-101: 77% (+1%).
ResNet-152: 78% (+1%).
ResNet-200: 78.3% (+0.3%).
ResNet-1001 (cifar): saturate.

Diminishing returns:

Accuracy gain per layer halves with depth.
Compute doubles, accuracy 0.3% gain — bad trade.
Overfitting concern at extreme depth.

Why deeper not always better:

Information bottleneck: early feature already rich.
Optimization difficulty: long path more local minima.
Compute/memory: training time prohibitive.
Inductive bias mismatch: very deep — task-specific need vary.

Better strategies than depth-only:

Width: Wide ResNet — same depth, wider channel.
Resolution: bigger input image.
Group conv: ResNeXt — same params, better capacity.
Attention: SE blocks — channel attention.
Compound scaling: EfficientNet — depth + width + resolution balanced.

EfficientNet-B7 (২০১৯):

Depth 813 layer equivalent (compound scaled).
84% top-1 — ResNet-152 (78%) থেকে অনেক ভালো।
Same FLOPs, different distribution.

Modern era:

"Deeper" ছেড়ে — "wider" বা "smarter" approach.
ConvNeXt: simple but tuned blocks.
Vision Transformer: attention based — different scaling.

Production reality:

ResNet-50 — most production deployment.
ResNet-101/152 — research, accuracy-critical.
200+ rare — diminishing return clear.

Bangladesh-এর context:

GPU limited — ResNet-50 sweet spot.
Mobile — MobileNet, EfficientNet-B0.
Server with budget — ResNet-101 good.

মূল উপলব্ধি: "Depth = capacity" — partial truth. Smart architecture trumps brute depth. ResNet's contribution — depth possible, কিন্তু pursue carefully.

প্র ০৪ Vision Transformer ResNet-কে replace করতে ক্রমশ পারছে। ২০২৬-এ ResNet-এর role কোথায়? "Legacy" নাকি "essential"?

এটি ২০২৬-এ active debate. Honest assessment — both/and rather than either/or.

ResNet still essential:

Mobile/edge: CNN inductive bias, fast inference.
Small dataset: ResNet pretrain transfers ভাল। ViT অনেক data চাই।
Detection: Faster R-CNN, YOLO — ResNet backbone dominant.
Medical imaging: regulatory acceptance, interpretability.
Industrial systems: millions of deployed models.

ViT taking over:

Large-scale image classification: JFT-pretrained ViT outperforms.
Multimodal: CLIP-style vision-language — ViT default.
Foundation models: DINOv2, MAE, EVA — ViT.
Generation: DiT (Diffusion Transformer) — Stable Diffusion 3.

Hybrid approaches winning:

ConvNeXt: CNN modernized with transformer tricks.
Swin: Window-based attention — CNN-like locality.
CoAtNet: Conv early, attention late.
EfficientFormer: Mobile-optimized hybrid.

Compute landscape:

ViT — GPU/TPU চাই, batch matters.
ResNet — CPU runnable, mobile chips optimize.
Bangladesh deployment — ResNet pragmatic.

Specific use case:

Image classification (ImageNet-style): ConvNeXt > ViT > ResNet.
Object detection: ResNet-50 + DINO/RT-DETR competitive.
Segmentation: Mask2Former (transformer) leading.
Self-supervised: ViT-MAE dominant.
Mobile: MobileNetV3, EfficientNet (CNN).

Future trajectory (5-year):

ResNet-50 — likely declining but persistent.
ConvNeXt-style updated CNN — surviving.
ViT — continuing growth in foundation model space.
New paradigms (Mamba, RWKV-vision) — potential disruption.

Skill investment recommendation:

Master ResNet — historical foundation, current production.
Master ViT — future-proof.
Understand hybrid — most practical work.
Don't bet on either alone.

মূল উপলব্ধি: Architecture-এ "winner takes all" rare. ResNet — Like SQL among databases — old, ubiquitous, still essential despite newer alternatives. Engineer-এর pragmatism: tool-fluent across paradigms.

অনুশীলন

Block params: ResNet-50 bottleneck (in=256, mid=64, out=256) — total params (BN ignored)?

$256 \times 64 + 64 \times 9 \times 64 + 64 \times 256 = 16{,}384 + 36{,}864 + 16{,}384 = 69{,}632$ ~70K.

Transfer learning: Pretrained ResNet-50 দিয়ে binary classification (cat/dog) — last layer modify.

m = resnet50(weights='DEFAULT')
# Freeze all except last
for p in m.parameters():
    p.requires_grad = False
m.fc = nn.Linear(2048, 2)  # binary head
# Now train m.fc only

ভাবুন: ResNet-এ skip connection-এর বদলে concatenation (DenseNet-style) করলে কী হত?

Channel-wise grow — each block channel double করতে বাধ্য। Memory overhead massive. ResNet add-এ — channel constant. DenseNet feature reuse-এ better (parameter কম)। Trade-off ভিন্ন।