Pooling — Max ও Avg

Aliasing — signal processing-এর ক্লাসিক সমস্যা। stride-2 sampling Nyquist criterion ভাঙে।

সমস্যাটি:

• Stride-2 pool — input-এর প্রতি ২য় position।
• Input ১ pixel shift করলে — pool-এর "even" position-গুলো "odd" হয়ে যায়।
• Output সম্পূর্ণ ভিন্ন — invariance fail।

Empirical verification:

• Zhang showed — image ১ pixel shift করলে CNN-এর top-1 prediction probability ৪০% case-এ বদলায়।
• Same image, slightly different translation → different class।
• Adversarial-এর কাছাকাছি behavior।

Fix — anti-aliased pooling:

• Stride-2 max pool-এর আগে blur (low-pass filter) — Nyquist preserve।
• Triangle filter বা Gaussian — typical।
• Performance + invariance দু'টোই improve।

BlurPool implementation:

class BlurPool(nn.Module):
    def __init__(self, channels, stride=2):
        super().__init__()
        kernel = torch.tensor([1., 2., 1.])
        kernel = kernel[:, None] * kernel[None, :]
        kernel = kernel / kernel.sum()
        self.register_buffer('kernel',
            kernel[None, None].repeat(channels, 1, 1, 1))
        self.stride = stride
    def forward(self, x):
        return F.conv2d(x, self.kernel,
                        stride=self.stride,
                        padding=1,
                        groups=x.shape[1])

Adoption:

• Antialiased ResNet — accuracy + robustness improve।
• Detection/Segmentation — bigger gain।
• Computational cost — minimal।

Other invariance approaches:

• Data augmentation: random crops train invariance imposed।
• Group equivariant CNN: rotation/scale invariance built-in।
• Transformer: patches different invariance properties।

Theoretical foundation:

• Sampling theorem — band-limited signal proper sampling।
• CNN-এ filtered বনাম unfiltered downsampling।
• Classical signal processing wisdom — DL-এ rediscovered।

Modern best practice:

• Stride-2 conv better than stride-2 pool (learnable filter)।
• Average pool — implicit smoothing।
• BlurPool — explicit anti-aliasing।
• Strong augmentation — data-driven invariance।

মূল উপলব্ধি: "Pooling = translation invariance" — partially true। Aliasing-এর জন্য actual invariance imperfect। BlurPool, stride-conv, augmentation — combined approach। Robustness-এ critical, especially safety-critical applications।

GAP — Lin et al. (২০১৪) "Network in Network"। CNN architecture-এ পরিবর্তন।

Traditional CNN head:

• Conv → ... → Flatten → FC(4096) → FC(4096) → FC(1000)।
• VGG-16 — শেষের ৩টি FC layer-এ ১২০M parameter (পুরো network-এর ৯০%)।
• Overfitting risk বিশাল।

GAP head:

• Conv → ... → GAP → FC(num_classes)।
• Parameter — শুধু final FC ($C \times \text{classes}$)।
• Spatial structure সম্পূর্ণ collapse।

Benefits:

• Massive parameter reduction: 90%+ less।
• Less overfitting: small data-এ critical।
• Input size flexibility: different resolution accept।
• Interpretability: CAM (Class Activation Map) সহজ।

CAM-এর secret sauce:

• Last conv feature map (before GAP) — semantic spatial info।
• FC weight × feature map = heatmap of class evidence।
• "Where is the model looking?" — visualization।

Limitation — spatial info loss:

• Object position information loss।
• Detection/segmentation-এ direct GAP কাজে লাগে না।
• Fine-grained classification — sometimes underperform।

Modern variants:

• Generalized Mean Pooling (GeM): $\left(\frac{1}{N}\sum x^p\right)^{1/p}$ — learnable $p$।
• Attention pool: learnable spatial weights।
• Concat pool: GAP + GMP concatenated।

Adoption history:

• NIN (২০১৪) introduce।
• GoogLeNet (২০১৪) — adopted।
• ResNet (২০১৫) — standard।
• Modern CNN — universal।

When NOT to use GAP:

• Object detection — spatial info needed (ROI pool)।
• Semantic segmentation — pixel-level prediction।
• Small input — Flatten + FC sometimes better।
• Fine-grained — bilinear pool variants।

মূল উপলব্ধি: GAP — DL architecture design-এর elegant innovation। Parameter কমে + interpretability + flexibility। Classification-এ universal। কিন্তু spatial task-এ alternative। "Less is more" — DL-এ powerful principle। Modern CNN — GAP without question।

Max pool — non-differentiable। কিন্তু DL-এ "subgradient" বা "selector" interpretation works।

Forward:

• Window-এ max value select।
• সেই position-এর index store (cache)।

Backward:

• Output gradient — শুধু max position-এ pass।
• অন্য position — gradient 0।
• "Hard selector" — discrete decision।

Implementation:

# PyTorch internal — pseudo
class MaxPoolBackward:
    def forward(self, x):
        self.indices = x.argmax_per_window()
        return x.max_per_window()

    def backward(self, grad_output):
        grad_input = zeros_like(x)
        grad_input.scatter_(self.indices, grad_output)
        return grad_input

Sparse gradient সমস্যা:

• 2x2 max pool — 75% neuron-এ gradient 0।
• "Dead" neurons — never update।
• Non-max neurons learning slow।

Avg pool comparison:

• Avg — সব 4 neuron-এ equal gradient (1/4)।
• Smoother backprop।
• সব neuron train পায়।

Soft alternatives:

• LogSumExp: $\log \sum e^x$ — smooth max approximation।
• Soft pool: weighted average — weight = $e^x / \sum e^x$।
• Stochastic pool: probability proportional to value।

Why max still works:

• Different windows — different position max।
• Aggregate — effective gradient flow।
• Empirical performance — strong।
• Sparse gradient — implicit regularization।

Implementation detail:

• Tied gradient between consecutive samples (cudnn cached indices)।
• Variable window for irregular input।
• Adaptive max pool — different output sizes।

Theoretical view:

• Subgradient — non-differentiable function-এর জন্য।
• Almost everywhere differentiable — measure-zero ties।
• Gradient method-এ acceptable।

Practical tips:

• Network depth-এ avg-এর চেয়ে max popular।
• Dead neuron occasional — large enough capacity।
• BN + ReLU — overall flow maintain।
• Average for smoothing, max for detection।

Recent insights:

• Vision Transformer — pooling avoid (different architecture)।
• Modern CNN — stride conv replaces max pool।
• Pure pool architecture rare আজ।

মূল উপলব্ধি: Max pool-এর gradient sparse — surprising kintu effective। Non-differentiable point-এ subgradient ব্যবহার। Soft alternatives exist কিন্তু practical performance gap small। Modern trend — stride conv বা pure attention। Classical max pool — historical importance + simplicity।

Medical imaging — high stakes। Pooling decision careful।

Domain considerations:

• X-ray: 1024×1024 typical resolution।
• TB lesion — small spatial features (cm scale)।
• Class imbalance: TB rare disease (5-10%)।
• Critical: false negative প্রাণহানি।

Architecture choice:

• Backbone: ResNet50 / DenseNet121 (medical-pretrained)।
• CheXNet (Stanford) — DenseNet121 — chest X-ray gold standard।
• Modern: Vision Transformer fine-tune।

Pooling strategy:

(১) Early — aggressive:

• Stem: 7×7 conv stride 2 + 3×3 max pool stride 2।
• 1024×1024 → 256×256 quickly।
• Compute reduction priority।
• Low-level features stable।

(২) Mid — conservative:

• Stride-2 conv between blocks।
• Max pool avoid (lesion ছোট)।
• Atrous/dilated conv consider।

(৩) Late — minimal:

• Final stage — 14×14 spatial keep।
• No additional pooling।
• Receptive field already large।

(৪) Head:

• Global Average Pooling।
• FC → binary (TB vs no TB)।
• Dropout 0.3-0.5 (small data)।

Modified for TB-specific:

class TBClassifier(nn.Module):
    def __init__(self, backbone='densenet121'):
        super().__init__()
        self.backbone = timm.create_model(
            backbone,
            pretrained=True,
            features_only=True,
        )
        # Multi-scale feature pyramid
        self.fpn = FeaturePyramidNetwork()
        self.gap = nn.AdaptiveAvgPool2d(1)
        self.head = nn.Sequential(
            nn.Dropout(0.4),
            nn.Linear(256, 2),
        )

Multi-scale strategy:

• FPN (Feature Pyramid Network) — multi-scale lesions।
• Different resolution combine।
• Small + large lesions detect।

Augmentation (Bangladesh climate):

• Brightness — exposure variation।
• Contrast — image quality differ।
• Random crops — orientation।
• Mixup — class imbalance address।

Validation strategy:

• Stratified k-fold।
• Site-wise hold-out (different hospital)।
• Sensitivity-specificity trade-off।
• FROC analysis — lesion localization।

Bangla deployment context:

• Rural clinic — low-resource hardware।
• Mobile app — quantized model।
• Offline inference — no cloud।
• Local language UI — actionable results।

Edge case:

• X-ray quality variation।
• Equipment difference (Siemens vs GE)।
• Chest size variation।
• Pediatric vs adult।

Regulatory:

• BMRC approval।
• Doctor in loop — never autonomous।
• Confidence calibration critical।
• Uncertainty quantification।

Resource-constrained tips:

• Knowledge distillation: ResNet50 → MobileNetV3।
• Quantization: INT8।
• ONNX runtime mobile।
• Edge device benchmark।

মূল উপলব্ধি: Medical imaging-এ pooling design — domain-specific। Aggressive early (compute), conservative mid (lesion), minimal late (receptive field already enough)। GAP standard head + dropout। Multi-scale FPN small lesion-এ। Bangladesh deployment — quantization + offline + Bangla UI। Doctor + AI partnership — patient safety।