ResNet — Skip Connection

Veit et al.-এর "Residual Networks Behave Like Ensembles of Relatively Shallow Networks" — ResNet-এর alternative theoretical understanding।

Ensemble interpretation:

• $n$ residual block — $2^n$ possible path।
• Each block — skip বা F(x) "binary choice"।
• প্রতিটি path — different network।
• Output — সব path-এর effective sum।

Mathematical expansion:

• $y_3 = y_2 + F_3(y_2) = (y_1 + F_2(y_1)) + F_3(y_1 + F_2(y_1))$।
• Recursive expand — $2^n$ sub-network-এর sum।
• Direct path (all skip) — input passthrough।
• Long path (all F) — full computation।

Empirical evidence:

• Test time block drop — small accuracy drop।
• Plain network — single block drop দেদার damage।
• Multiple block drop — gradual degradation।

Path length distribution:

• Most paths — relatively short (5-17 blocks)।
• Long paths — exist but contribute less।
• Effective depth small।

Stochastic Depth (Huang ২০১৬):

• Train-এ randomly block drop।
• Test-এ full network।
• Implicit shorter ensemble।
• 1200-layer ResNet train possible।

Implementation:

class StochasticBlock(nn.Module):
    def __init__(self, in_ch, out_ch, drop_p=0.0):
        super().__init__()
        self.block = BasicBlock(in_ch, out_ch)
        self.drop_p = drop_p

    def forward(self, x):
        if self.training and torch.rand(1) < self.drop_p:
            return x  # skip block
        return self.block(x)

Implications:

• "Deep" misnomer — effectively shallow ensemble।
• Path diversity learning-এ key।
• Architecture-এ "hidden" parallel computation।
• Training dynamics — multi-scale interaction।

Theoretical implications:

• Generalization bound — depth less important।
• Width vs depth trade-off।
• Wide ResNet (shallower wider) competitive।
• Architecture exploration insight।

Modern relevance:

• EfficientNet — scaling balance based on this insight।
• Transformer — similar residual ensemble structure।
• NAS — block-wise search natural।

Counter-argument:

• "Pure ensemble" overstates।
• Path interaction matters।
• Joint training — pure ensemble different।
• Empirical limit।

Practical implications:

• ResNet robust — block fail tolerable।
• Transfer learning effective।
• Layer pruning practical।
• Stochastic depth regularization।

Bangladesh edge deployment:

• Block pruning model size reduce।
• Adaptive inference compute save।
• Mobile-friendly variants।
• Quality-cost trade-off।

মূল উপলব্ধি: ResNet — single deep network deceptive। Actually $2^n$ shallow path ensemble। Skip connection block-level ensemble enable। Stochastic depth regularization extension। Modern architecture — ensemble interpretation pervasive। Practical pruning, robustness, transfer learning insight। DL theory-র elegant alternative perspective।

He et al.-এর follow-up paper "Identity Mappings in Deep Residual Networks" — ResNet design refinement।

Original ResNet block:

• Conv → BN → ReLU → Conv → BN → ADD → ReLU
• BN-ReLU after conv (post-activation)।
• Final ReLU after sum।

Pre-activation version:

• BN → ReLU → Conv → BN → ReLU → Conv → ADD
• BN-ReLU before conv (pre-activation)।
• No ReLU after sum।
• Identity path completely clean।

Why "identity matters":

• Original — sum-এর পরে ReLU।
• ReLU non-linear — identity path এ "interferes"।
• Pre-activation — sum after conv, no ReLU।
• Pure additive identity flow।

Mathematical advantage:

• $y = x + F(x)$ exactly।
• Gradient: $\frac{\partial L}{\partial x} = \frac{\partial L}{\partial y} \cdot (1 + \frac{\partial F}{\partial x})$।
• The "1" — pure identity path, never blocked।
• Original-এ ReLU sometimes derivative 0।

Empirical results:

• ResNet-1001 — pre-activation যথেষ্ট train।
• ResNet-1001 (post-activation) — diverge।
• 200-layer-এ ~1% accuracy gain।
• Deeper network easier।

Implementation:

class PreActBlock(nn.Module):
    def __init__(self, in_ch, out_ch, stride=1):
        super().__init__()
        self.bn1   = nn.BatchNorm2d(in_ch)
        self.conv1 = nn.Conv2d(in_ch, out_ch, 3,
                               stride, 1, bias=False)
        self.bn2   = nn.BatchNorm2d(out_ch)
        self.conv2 = nn.Conv2d(out_ch, out_ch, 3,
                               1, 1, bias=False)
        self.shortcut = (
            nn.Conv2d(in_ch, out_ch, 1, stride, bias=False)
            if (stride != 1 or in_ch != out_ch)
            else nn.Identity()
        )

    def forward(self, x):
        out = F.relu(self.bn1(x), inplace=True)
        shortcut = self.shortcut(out if not isinstance(self.shortcut, nn.Identity) else x)
        out = self.conv1(out)
        out = F.relu(self.bn2(out), inplace=True)
        out = self.conv2(out)
        return out + shortcut

Comparison summary:

• Post-activation (original): ReLU(x + F(x))
• Pre-activation: x + F(x), F has BN-ReLU first
• Cleaner identity path
• Better very-deep training

Why post-activation worked too:

• 50-152 layer — ReLU effect minor।
• Deeper — accumulate problem।
• Most production ResNet still post-activation।

Modern adoption:

• 1000-layer experiments — pre-activation।
• Standard 50-152 — post-activation।
• Both pre-trained available।
• Choice depends on depth।

Transformer parallel:

• Pre-LN Transformer (newer)।
• Post-LN Transformer (original)।
• Same gradient flow consideration।
• Pre-LN training stability।

Practical recommendations:

• Standard depth — original sufficient।
• Very deep experimentation — pre-activation।
• NAS — both options consider।
• Domain-specific tune।

Bangladesh deployment:

• Standard ResNet sufficient most cases।
• Custom training scenarios — pre-activation।
• Pre-trained both available।
• Architecture detail rarely matters production।

মূল উপলব্ধি: Pre-activation ResNet — refinement focused on gradient flow। Identity path কোনো interference নেই। Very deep network train enable। Production-এ rare necessity but theoretical important। Transformer-এ similar pattern (Pre-LN/Post-LN)। Architecture detail matters scaling extreme। Standard usage — original sufficient।

Skip connection — ML history-র সবচেয়ে impactful single idea-গুলোর একটি। Simple কিন্তু transformative।

Pre-skip era সমস্যা:

• Vanishing gradient — sigmoid/tanh deep network।
• Optimization difficulty — even 30 layer struggle।
• BN আশাকাল করল — কিন্তু 100+ layer impossible।
• Architecture barrier।

Skip connection ফল:

• 1000+ layer network train possible।
• ImageNet error 3% (human-level)।
• Detection, segmentation — সব advance।
• Foundation model era enable।

Cross-architecture impact:

• Transformer: "x + Attention(x), x + FFN(x)" — fundamental।
• U-Net: encoder-decoder skip — segmentation revolution।
• DenseNet: all-to-all extreme version।
• Wide ResNet, ResNeXt, EfficientNet: সব residual।

Why "small" addition transformative:

• Optimization landscape smooth।
• Identity mapping easy learn।
• Gradient direct flow।
• Training dynamics stable।

The minimal change argument:

• Pre-ResNet block: $y = F(x)$।
• Post-ResNet block: $y = F(x) + x$।
• Single addition operation।
• No new learnable parameter।
• Compute negligible increase।

Theoretical perspective:

• Optimization: identity easy reach।
• Generalization: implicit regularization।
• Ensemble: exponential path।
• Information flow: bypass available।

Beyond DL:

• Highway networks (Schmidhuber, ২০১৫) — gated version।
• LSTM cell state — temporal skip।
• Recurrent skip — sequence model।
• Idea pervasive across architecture।

Industrial transformation:

• Pre-ResNet — DL niche application।
• Post-ResNet — every CV task।
• Foundation model basis।
• Modern AI infrastructure।

Hinton's quote:

• "The most important idea in deep learning"।
• (Across many talks)।
• Not exaggeration।
• Skip connection central role।

Modern variations:

• Dense: all-to-all।
• Multi-scale: different resolution।
• Attention residual: Transformer।
• Gated: learnable weights।

Why was it "missed" so long:

• Field's depth focus — kept adding layers।
• Hindsight obvious — পেতে কঠিন।
• Empirical observation pre-ResNet।
• Theoretical understanding post-hoc।

Lesson for researchers:

• Simple ideas powerful।
• Question fundamental assumption।
• Architecture detail matters।
• Theory follow practice।

Bangladesh implication:

• Modern model adopt — skip standard।
• Training stability default।
• Transfer learning effective।
• Local task — modern architecture।

মূল উপলব্ধি: Skip connection — DL-এর foundational breakthrough। Single addition operation, transformative impact। Optimization, generalization, gradient flow — সব address। Modern architecture-এ universal। Bangladesh — modern model standard usage, skip implicit। ML innovation — sometimes simplest idea most powerful। ResNet legacy — eternal।

Medical imaging + small data + ResNet-50 — practical Bangladesh scenario। Strategy carefully balance।

Pre-trained model selection:

• ImageNet pretrained — natural image, transferable।
• RadImageNet — medical pretrained — better।
• CheXNet — DenseNet121, chest X-ray specific।
• Best: domain-pretrained + ImageNet।

Transfer learning options:

• (1) Feature extraction: freeze all + new head।
• (2) Fine-tune top: freeze early + train last layers।
• (3) Full fine-tune: all weights train, low lr।
• (4) Discriminative lr: different lr per layer।

5000 sample - approach 2 ভাল:

import torchvision.models as m

model = m.resnet50(weights='IMAGENET1K_V2')

# Replace head
model.fc = nn.Sequential(
    nn.Dropout(0.5),
    nn.Linear(2048, 14),  # 14 disease class
)

# Freeze early layers
for p in model.layer1.parameters():
    p.requires_grad = False
for p in model.layer2.parameters():
    p.requires_grad = False

# Train layer3, layer4, fc
trainable = [p for p in model.parameters() if p.requires_grad]
optim = torch.optim.AdamW(trainable, lr=1e-4, weight_decay=0.01)

Discriminative learning rates:

params = [
    {'params': model.layer3.parameters(), 'lr': 1e-5},
    {'params': model.layer4.parameters(), 'lr': 1e-4},
    {'params': model.fc.parameters(),     'lr': 1e-3},
]
optim = torch.optim.AdamW(params, weight_decay=0.01)

Augmentation strategy:

train_transform = A.Compose([
    A.Resize(256, 256),
    A.RandomCrop(224, 224),
    A.HorizontalFlip(p=0.5),  # X-ray দু'দিকে symmetric
    A.RandomRotation(10),
    A.ColorJitter(0.1, 0.1, 0.1),
    A.GaussianBlur(blur_limit=3, p=0.3),
    A.GaussNoise(p=0.3),
    A.Normalize(),
    ToTensorV2(),
])
# Test transform: শুধু resize + center crop + normalize

Class imbalance handling:

• TB rare (~5-10%)।
• Weighted CrossEntropyLoss।
• Focal loss alternative।
• SMOTE for minority class।

Training schedule:

• Stage 1: head only train (5 epoch)।
• Stage 2: top blocks unfreeze (15 epoch)।
• Stage 3: full unfreeze low lr (10 epoch)।
• Cosine annealing lr schedule।

Regularization:

• Dropout 0.5 in head।
• Weight decay 0.01।
• Mixup augmentation।
• Label smoothing 0.1।

Validation strategy:

• 5-fold cross-validation।
• Stratified by disease label।
• Hold-out test from different hospital।
• Bangladesh-specific test set।

Bangladesh-specific challenges:

• X-ray equipment variation (Siemens, GE, locally manufactured)।
• Image quality difference।
• Patient population — different baseline।
• Radiologist labeling consistency।

Domain adaptation:

• Synthetic data — local hospital simulation।
• Test-time augmentation।
• Adversarial training।
• Self-supervised pretrain on unlabeled BD X-rays।

Ensemble strategy:

• 5 models from k-fold।
• Test-time augmentation।
• Model averaging।
• ~2-3% accuracy gain।

Interpretability:

• Grad-CAM visualization।
• Attention map।
• Doctor verification।
• Explainable predictions।

Calibration:

• Probability calibrate (temperature scaling)।
• Uncertainty quantification (MC Dropout)।
• Confidence threshold।
• Doctor-in-loop borderline।

Deployment:

• ONNX export।
• Quantization (INT8)।
• Mobile deployment (rural clinic)।
• Bangla report generation।

Active learning:

• Uncertain samples — expert label।
• Iterate dataset growth।
• Cost-effective improvement।

Validation metrics:

• AUC-ROC per class।
• F1 score।
• Sensitivity-specificity।
• NPV/PPV (clinical context)।

Realistic expectations:

• 5000 sample — 80-85% accuracy।
• Comparable to junior radiologist।
• Senior radiologist ground truth।
• Continuous improvement।

Regulatory:

• BMRC (Bangladesh Medical Research Council) approval।
• Doctor-in-loop mandatory।
• Liability framework।
• Patient consent।

মূল উপলব্ধি: Medical imaging fine-tune — careful strategy। Pretrained foundation + augmentation + class balance + validation। ResNet-50 strong baseline। Discriminative lr + staged unfreezing optimal। Bangladesh — domain adaptation + Bangla integration + regulatory compliance। 5000 sample challenging but achievable। Iterative improvement via active learning। Doctor-AI partnership clinical safety।