পাঠ ১৪ · ৩৫-এর মধ্যে · মডিউল ২

Home / AI Courses / Computer Vision / ResNet

ResNet — skip connection-এর শক্তি

ResNet — residual learning, the deepest practical CNN

৮ মিনিট পড়া মাঝারি · Intermediate PyTorch কোডসহ

এই পাঠে যা শিখবেন

Residual learning-এর গাণিতিক intuition
Basic block ও bottleneck block-এর architecture
ResNet-18, 34, 50, 101, 152 — কোনটা কখন
PyTorch-এ ResNet implementation

১ · Degradation problem (revisit)

VGG-এর experiment — 19 layer-এর পরে accuracy কমে। Counter-intuitive — capacity বাড়লে কেন accuracy কম?

He et al.-এর observation: "যদি deeper network-এর extra layer identity হত — তাহলে অন্তত equal accuracy থাকা উচিত। Worse মানে — optimization fail।"

কেন্দ্রীয় ধারণা

Identity mapping বানানো কঠিন non-linear network-এ। তাই — identity-কে easy default বানিয়ে নিন। "Residual" শেখাতে দিন — পার্থক্যটুকু।

২ · Residual learning

Standard CNN block: input $x$ → $F(x)$ output।

ResNet block:

$$y = F(x) + x$$

$F(x)$ = Conv-BN-ReLU-Conv-BN। "Residual"।
$x$ — direct skip connection।
Output $y$ — both summed।

Key insight: network-এর শেখা কাজ এখন $F(x)$ — যা residual। Identity যদি optimal — $F(x) = 0$ শেখা সহজ (just zero out the weights)।

৩ · Why does it work?

Gradient flow: backprop-এ skip path-এর gradient direct early layer-এ পৌঁছে। Vanishing gradient problem dissolved।
Smoother loss landscape: Li et al. (২০১৮) visualization — ResNet loss surface অনেক বেশি smooth।
Implicit ensemble: $2^n$ paths through n blocks — multiple sub-networks ensemble।
Easy identity: degradation problem direct address।

৪ · Basic block (ResNet-18, 34)

x → Conv 3×3 → BN → ReLU → Conv 3×3 → BN → (+) → ReLU → output
       └─────────── skip ─────────────────────┘

Two 3×3 conv + skip। ResNet-18 ও 34-এ ব্যবহৃত।

৫ · Bottleneck block (ResNet-50, 101, 152)

x → Conv 1×1 (compress) → BN → ReLU
  → Conv 3×3 (process)  → BN → ReLU
  → Conv 1×1 (expand)   → BN → (+) → ReLU → output
   └─────────────── skip ──────────────┘

Inception-এর bottleneck idea: 1×1 → 3×3 → 1×1। Compute সাশ্রয়। 256 → 64 → 64 → 256 typical।

৬ · ResNet variants

Variant	Block type	Params	Top-1
ResNet-18	Basic	11.7M	69.8%
ResNet-34	Basic	21.8M	73.3%
ResNet-50	Bottleneck	25.6M	76.1%
ResNet-101	Bottleneck	44.5M	77.4%
ResNet-152	Bottleneck	60.2M	78.3%

ResNet = "Don't reinvent identity, just learn what to add to it." একজন editor-এর মতো — original draft preserve, শুধু changes mark। Engineering elegance-এর peak।

ResNet block — residual transformation $F(x)$ + identity skip। Bottleneck-এ 1×1 দিয়ে compute সাশ্রয়।

৭ · PyTorch-এ ResNet block

Python · PyTorch

import torch
import torch.nn as nn

class BasicBlock(nn.Module):
    def __init__(self, ch):
        super().__init__()
        self.conv1 = nn.Conv2d(ch, ch, 3, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(ch)
        self.conv2 = nn.Conv2d(ch, ch, 3, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(ch)
        self.relu = nn.ReLU(inplace=True)
    def forward(self, x):
        identity = x
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out = out + identity                # residual add
        return self.relu(out)

block = BasicBlock(64)
x = torch.randn(1, 64, 56, 56)
y = block(x)
print("In:", x.shape, "Out:", y.shape)

# Pretrained ResNet-50
from torchvision.models import resnet50, ResNet50_Weights
m = resnet50(weights=ResNet50_Weights.IMAGENET1K_V2)
n = sum(p.numel() for p in m.parameters())
print(f"ResNet-50 params: {n:,}")           # ~25.5M

Residual block out + identity — মাত্র এই একটি line CV-কে বদলে দিয়েছে। torchvision-এর ResNet-50 — ImageNet pretrained, transfer learning-এর gold standard।

৮ · ResNet-এর variants

ResNet v2 (২০১৬, He et al.): pre-activation — BN-ReLU আগে, conv পরে। Slightly better।
ResNeXt (২০১৭): grouped convolution। 32-group, similar params, +1% accuracy।
Wide ResNet: deep-এর বদলে wider — 50-layer wider better than 152 thin।
Squeeze-and-Excitation ResNet (SENet): channel attention add। ImageNet 2017 winner।

৯ · Modern uses

Object detection: Faster R-CNN, Mask R-CNN — ResNet-50/101 backbone।
Segmentation: FCN, DeepLab, U-Net++ — ResNet encoder।
Pose estimation: HRNet — ResNet-derived।
Self-supervised: MoCo, SimCLR, BYOL — ResNet-50 backbone।
Medical: chest X-ray, retinal — ResNet-50 transfer learning।
Style transfer: ResNet feature also use।

ResNet AI-র সবচেয়ে cited paper-এর একটি (১৬০K+ citations)। ২০২৬-এ এসেও — production CNN-এর majority ResNet backbone। Mastery essential।

ভাবনার প্রশ্ন

প্র ০১ ResNet-এর skip connection identity-add। কিন্তু channel mismatch (64 → 128) হলে কী হয়? Spatial size mismatch হলে?

এটি ResNet implementation-এর crucial detail। Naive skip every-block-এ কাজ করে না।

Channel mismatch:

ResNet-এ stage transition-এ channel double (64 → 128 → 256 → 512)।
Direct add impossible — different channel count।
Solution: 1×1 conv on skip path — channel project।

Spatial mismatch:

Stage transition-এ 2x downsample (stride=2)।
Skip-ও same downsample চাই।
Solution: 1×1 stride-2 conv on skip path।

Two skip variants:

# Identity skip (when shapes match)
out = F(x) + x

# Projection skip (when shapes mismatch)
out = F(x) + Conv1x1_stride2(x)

Original paper-এ option (A):

Zero-padding for extra channels।
Spatial — every-other element subsample।
No extra parameter।
Slightly worse accuracy।

Option (B): Projection only at downsample (default):

Within-stage skip = identity।
Between-stage skip = 1×1 conv projection।
Standard ResNet implementation।

Option (C): Projection always:

Every block-এ 1×1 projection।
Marginal accuracy gain।
Param overhead তুলনামূলক বেশি।

Code structure:

class ResBlock(nn.Module):
    def __init__(self, in_ch, out_ch, stride=1):
        super().__init__()
        self.conv1 = nn.Conv2d(in_ch, out_ch, 3, stride, 1)
        self.conv2 = nn.Conv2d(out_ch, out_ch, 3, 1, 1)

        # Projection skip if needed
        if stride != 1 or in_ch != out_ch:
            self.skip = nn.Conv2d(in_ch, out_ch, 1, stride)
        else:
            self.skip = nn.Identity()

Pre-activation vs post-activation:

Original: Conv-BN-ReLU-Conv-BN + skip → ReLU।
v2: BN-ReLU-Conv-BN-ReLU-Conv + skip।
v2 — gradient flow better, slightly higher accuracy।

Modern implementations:

torchvision ResNet — option (B)।
Timm library — multiple variants।
Custom paper — task-specific tweaks।

মূল উপলব্ধি: "Skip connection trivial" misconception। Implementation-এ subtle, performance impact noticeable। Engineering attention to detail।

প্র ০২ "Implicit ensemble" hypothesis — ResNet-কে $2^n$ paths-এর ensemble বলে দেখা যায়। এই interpretation মাথায় কী implication?

Veit et al. (২০১৬, NeurIPS) — "Residual Networks Behave Like Ensembles"। সর্বাধিক influential interpretation।

Path count:

Each ResNet block — দু'টি option: take F(x) ও skip-only।
$n$ blocks → $2^n$ unique paths।
ResNet-152 — $2^{50}+$ paths!

Empirical evidence:

Random block skip — accuracy gentle drop।
VGG block skip — catastrophic drop।
ResNet — robust to layer ablation।

Length distribution of effective paths:

Path length distribution — binomial।
Most paths length n/2 (15-25 for ResNet-50)।
Very short ও very long paths rare।
Effective depth অনেক কম than nominal depth।

Implications:

Vanishing gradient: short path-এ direct gradient — primary value।
Stochastic depth: randomly skip block during training (DropPath)। Same intuition।
Layer pruning: some block remove without breaking — production-এ smaller model।

Stochastic Depth (Huang et al., 2016):

Training-এ 50% probability skip a block।
Inference-এ — সব block।
Acts as regularization + ensemble।
Train ResNet-1202!

Modern extension:

DropPath: ConvNeXt, Swin-এ standard।
RegNetY: path-level regularization।
NFNet: Normalizer-Free + scaled residual।

Theoretical depth:

Greff et al. — "highway/ResNet performs iterative refinement"।
Each block — small refinement to running representation।
Different from "deep abstraction" classical view।

Counter-evidence (Wu et al., 2018):

ResNet-152 reduce করে depth — accuracy drops।
"Pure ensemble" view incomplete।
Block-level interaction matters।

মূল উপলব্ধি: Network-এর behavior often defies architectural intent। ResNet "deep" শোনায় কিন্তু "ensemble" আচরণ। Empirical understanding-ই ground truth।

প্র ০৩ ResNet-152 ImageNet-এ 78%। ResNet-200, 1000 — কেন popular না? Diminishing returns কোথায়?

এটি architecture scaling-এর fundamental observation।

ResNet depth vs accuracy:

ResNet-18: 70%।
ResNet-34: 73% (+3%)।
ResNet-50: 76% (+3%)।
ResNet-101: 77% (+1%)।
ResNet-152: 78% (+1%)।
ResNet-200: 78.3% (+0.3%)।
ResNet-1001 (cifar): saturate।

Diminishing returns:

Accuracy gain per layer halves with depth।
Compute doubles, accuracy 0.3% gain — bad trade।
Overfitting concern at extreme depth।

Why deeper not always better:

Information bottleneck: early feature already rich।
Optimization difficulty: long path more local minima।
Compute/memory: training time prohibitive।
Inductive bias mismatch: very deep — task-specific need vary।

Better strategies than depth-only:

Width: Wide ResNet — same depth, wider channel।
Resolution: bigger input image।
Group conv: ResNeXt — same params, better capacity।
Attention: SE blocks — channel attention।
Compound scaling: EfficientNet — depth + width + resolution balanced।

EfficientNet-B7 (২০১৯):

Depth 813 layer equivalent (compound scaled)।
84% top-1 — ResNet-152 (78%) থেকে অনেক ভালো।
Same FLOPs, different distribution।

Modern era:

"Deeper" ছেড়ে — "wider" বা "smarter" approach।
ConvNeXt: simple but tuned blocks।
Vision Transformer: attention based — different scaling।

Production reality:

ResNet-50 — most production deployment।
ResNet-101/152 — research, accuracy-critical।
200+ rare — diminishing return clear।

Bangladesh-এর context:

GPU limited — ResNet-50 sweet spot।
Mobile — MobileNet, EfficientNet-B0।
Server with budget — ResNet-101 good।

মূল উপলব্ধি: "Depth = capacity" — partial truth। Smart architecture trumps brute depth। ResNet's contribution — depth possible, কিন্তু pursue carefully।

প্র ০৪ Vision Transformer ResNet-কে replace করতে ক্রমশ পারছে। ২০২৬-এ ResNet-এর role কোথায়? "Legacy" নাকি "essential"?

এটি ২০২৬-এ active debate। Honest assessment — both/and rather than either/or।

ResNet still essential:

Mobile/edge: CNN inductive bias, fast inference।
Small dataset: ResNet pretrain transfers ভাল। ViT অনেক data চাই।
Detection: Faster R-CNN, YOLO — ResNet backbone dominant।
Medical imaging: regulatory acceptance, interpretability।
Industrial systems: millions of deployed models।

ViT taking over:

Large-scale image classification: JFT-pretrained ViT outperforms।
Multimodal: CLIP-style vision-language — ViT default।
Foundation models: DINOv2, MAE, EVA — ViT।
Generation: DiT (Diffusion Transformer) — Stable Diffusion 3।

Hybrid approaches winning:

ConvNeXt: CNN modernized with transformer tricks।
Swin: Window-based attention — CNN-like locality।
CoAtNet: Conv early, attention late।
EfficientFormer: Mobile-optimized hybrid।

Compute landscape:

ViT — GPU/TPU চাই, batch matters।
ResNet — CPU runnable, mobile chips optimize।
Bangladesh deployment — ResNet pragmatic।

Specific use case:

Image classification (ImageNet-style): ConvNeXt > ViT > ResNet।
Object detection: ResNet-50 + DINO/RT-DETR competitive।
Segmentation: Mask2Former (transformer) leading।
Self-supervised: ViT-MAE dominant।
Mobile: MobileNetV3, EfficientNet (CNN)।

Future trajectory (5-year):

ResNet-50 — likely declining but persistent।
ConvNeXt-style updated CNN — surviving।
ViT — continuing growth in foundation model space।
New paradigms (Mamba, RWKV-vision) — potential disruption।

Skill investment recommendation:

Master ResNet — historical foundation, current production।
Master ViT — future-proof।
Understand hybrid — most practical work।
Don't bet on either alone।

মূল উপলব্ধি: Architecture-এ "winner takes all" rare। ResNet — Like SQL among databases — old, ubiquitous, still essential despite newer alternatives। Engineer-এর pragmatism: tool-fluent across paradigms।

অনুশীলন

Block params: ResNet-50 bottleneck (in=256, mid=64, out=256) — total params (BN ignored)?

$256 \times 64 + 64 \times 9 \times 64 + 64 \times 256 = 16{,}384 + 36{,}864 + 16{,}384 = 69{,}632$ ~70K।

Transfer learning: Pretrained ResNet-50 দিয়ে binary classification (cat/dog) — last layer modify।

m = resnet50(weights='DEFAULT')
# Freeze all except last
for p in m.parameters():
    p.requires_grad = False
m.fc = nn.Linear(2048, 2)  # binary head
# Now train m.fc only

ভাবুন: ResNet-এ skip connection-এর বদলে concatenation (DenseNet-style) করলে কী হত?

Channel-wise grow — each block channel double করতে বাধ্য। Memory overhead massive। ResNet add-এ — channel constant। DenseNet feature reuse-এ better (parameter কম)। Trade-off ভিন্ন।