Transfer Learning

Yosinski et al. — "How transferable are features in deep neural networks?" — DL community-র classic paper। Multiple insight।

Findings:

• Layer 1-2 — task-independent features।
• Layer 3-4 — partial dependent।
• Layer 5+ — task-specific।
• Final layers — fully specific।

Co-adaptation problem:

• Network train-এ — layer parameters jointly optimized।
• Layer 3 weight depends on layer 2 weight।
• Layer 4 — layer 3 specific feature expect।
• Adjacent layers — codependent।

Why transfer breaks co-adaptation:

• Layer 1-3 transfer + layer 4-5 random initialize।
• Layer 4-5 — layer 3-এর "expected" pattern না দেখে।
• Co-adaptation broken।
• Optimization difficulty।

Empirical observation:

• Layer 3 only freeze + 4-5 random — accuracy drop।
• Layer 1-3 freeze + 4-5 fine-tune — better।
• Layer 1-7 freeze + 8 fine-tune — best for similar task।

The "splitting" experiment:

• Train AnB-এ — A, B random class split।
• BnB — same dataset transfer (no domain gap)।
• BnB-এ কিছু performance drop — pure co-adaptation ভাঙা।
• Even same-task transfer cost।

Practical implications:

• Feature extraction (freeze all but head) — co-adaptation preserved।
• Fine-tuning (all layers) — co-adaptation reform।
• Freezing partial — risky middle ground।
• Discriminative LR — best of both।

Modern understanding:

• BatchNorm — adaptation help।
• Skip connection — robust to layer shift।
• Self-attention — flexible adaptation।
• Modern transfer easier than 2014।

Adapter modules:

• Frozen pretrained + small trainable bottleneck।
• Co-adaptation preserved।
• New adaptation in small modules।
• Parameter-efficient (LoRA, Houlsby adapters)।

Bangladesh transfer scenarios:

• ImageNet → Bangla characters — full fine-tune (large gap)।
• ImageNet → Bangla flora — feature extraction enough।
• ImageNet → medical X-ray — discriminative LR।
• Domain matters।

মূল উপলব্ধি: Yosinski "transfer features" আবিষ্কার + co-adaptation challenge identify। Modern era — adapters, LoRA, BN — co-adaptation impact reduce। Feature extraction safe, fine-tune adjustable। Bangladesh-এ — domain gap-অনুসারে strategy। Transfer learning theory practical insight।

ImageNet — natural object centric। Medical/satellite — significantly different। Transfer learning সব domain-এ trivial না।

ImageNet bias:

• Center-cropped photo।
• Single object focus।
• Natural lighting।
• Real-world objects।
• 3-channel RGB।

Medical imaging-এ challenge:

• Grayscale প্রায়ই (X-ray)।
• Anatomical structure — natural object না।
• Subtle pathology — careful feature।
• Image quality variation hospital-অনুসারে।

Satellite imagery challenge:

• Top-down view।
• Multi-spectral (RGB+IR)।
• Object scale 1px - meter।
• Geographical pattern।

Empirical findings:

• Medical: ImageNet pretrain — slight benefit।
• RadImageNet (medical) — better।
• Domain pretrain — best।
• Self-supervised on domain data — competitive।

The "universal" myth:

• Universal — degree-অনুসারে।
• Edge/texture — most universal।
• Mid-features — somewhat।
• High-level — least।

Domain-specific pretraining:

• RadImageNet: 1.4M radiology images — medical features।
• BigEarthNet: 590K satellite images।
• BraTS: brain MRI segmentation।
• Domain pretrain consistently helpful।

Self-supervised alternatives:

• Unlabeled domain data utilize।
• Contrastive (SimCLR, MoCo)।
• Masked autoencoder (MAE)।
• DINOv2 — general-purpose।

Hybrid approach:

• ImageNet → domain pretrain → task fine-tune।
• Stagewise transfer।
• Each stage knowledge accumulate।
• Best results।

Bangladesh medical AI:

• CheXNet (DenseNet pretrained)।
• BdLekha medical text।
• Self-supervised on Bangladesh hospital data।
• Domain adaptation critical।

Practical strategy decision tree:

• Domain similar to ImageNet? → Standard transfer।
• Different domain, large data? → Self-supervised + fine-tune।
• Different + small data? → Domain pretrain + fine-tune।
• Very specialized? → Custom architecture + domain pretrain।

Cross-modal transfer:

• Audio spectrogram → image (works!)।
• 3D MRI → 2D ResNet (slice-wise)।
• Multi-spectral → RGB equivalent।
• Creativity in adaptation।

Common pitfalls:

• Wrong normalization (ImageNet stats on medical)।
• Standard augmentation may damage subtle pathology।
• Class imbalance handling।
• Validation strategy domain-specific।

Recent trends:

• Foundation models (DINOv2) — domain transfer good।
• Vision Transformer — adapt easier।
• Multi-modal pretraining।
• Few-shot learning।

Cost-benefit:

• Domain pretraining expensive।
• Sometimes ImageNet sufficient।
• Empirical comparison essential।
• Bangladesh — start simple।

মূল উপলব্ধি: ImageNet — natural image bias। Medical/satellite-এ partial transfer। Domain pretrain consistently helpful। Self-supervised emerging powerful। Bangladesh — Bengali characters, X-ray, satellite — domain-specific approach। Universal feature partial reality। Transfer learning landscape evolving fast।

Self-supervised learning (SSL) — deep learning-এর recent বিপ্লব। Labeled data-এর dependency drastically কমায়।

Supervised pretraining limit:

• Need labeled data (expensive)।
• ImageNet bias inherits।
• Bangla, Bangladesh labeled data scarce।
• Foundation model unlabeled use।

SSL key idea:

• Pretext task — auto-generated label।
• No human annotation।
• Massive unlabeled data utilize।
• General feature learn।

Major SSL methods:

(১) Contrastive (SimCLR, MoCo):

• একই image-এর দু'টো aug — pull together।
• ভিন্ন image — push apart।
• InfoNCE loss।
• Strong feature learn।

(২) Masked image modeling (MAE):

• BERT vision counterpart।
• 75% patch mask।
• Reconstruct missing।
• ViT-friendly।

(৩) Self-distillation (DINO, DINOv2):

• Student-teacher network।
• Different views — same feature predict।
• No negative samples।
• Stable training।

(৪) Predictive (BYOL, SwAV):

• One view — predict other view।
• Cluster assignment।
• Negative-free।

SimCLR architecture:

def simclr_loss(z1, z2, tau=0.1):
    """z1, z2: two augmented views' embeddings"""
    z = torch.cat([z1, z2], dim=0)  # (2N, D)
    z = F.normalize(z, dim=1)
    sim = z @ z.T / tau  # (2N, 2N)

    N = z1.size(0)
    targets = torch.arange(N).repeat(2)
    targets = (targets + N) % (2 * N)

    mask = torch.eye(2*N, dtype=torch.bool).to(z.device)
    sim.masked_fill_(mask, -float('inf'))

    return F.cross_entropy(sim, targets)

DINO key insight:

• Teacher EMA student।
• Centering + sharpening prevent collapse।
• Self-attention emergent properties।
• Object boundary auto-discovery।

MAE simplicity:

# MAE pseudocode
def forward(self, x):
    # Mask 75% patches
    x_masked, mask = random_mask(x, ratio=0.75)
    # Encode visible patches
    encoded = encoder(x_masked)
    # Decoder reconstruct full image
    decoded = decoder(encoded, mask)
    # Loss only on masked
    loss = F.mse_loss(decoded[mask], x[mask])
    return loss

Performance comparison:

• SimCLR — ResNet-50 on ImageNet — 76% (vs supervised 76%)।
• DINOv2 — ViT-L — 86%।
• MAE — ViT-H — 87%।
• Match/exceed supervised।

Transfer ability:

• SSL features — better generalization।
• Fewer labels need fine-tune।
• Robust to distribution shift।
• Few-shot strong।

Foundation model era:

• DINOv2 — universal vision feature।
• SAM — segmentation foundation।
• CLIP — image-text joint।
• Multi-task capable।

Bangladesh utilization:

• Unlabeled Bangladesh medical images abundant।
• SSL pretrain on local data।
• Few-shot Bangla character recognition।
• Cultural-specific feature emergence।

Compute requirements:

• SSL pretraining — expensive (8-32 GPU)।
• Fine-tune — affordable।
• Foundation model — academic infrastructure প্রয়োজন।
• Pretrained available (Hugging Face)।

Practical workflow:

• Start: pre-trained DINOv2/MAE।
• Fine-tune on labeled data।
• Few-shot if data scarce।
• Linear probe simple tasks।

Limitation:

• Pretext task not always task-relevant।
• Augmentation choice critical।
• Mode collapse risk।
• Foundation model bias inherits।

Future direction:

• Multi-modal SSL।
• Video SSL।
• 3D SSL।
• Domain-specific foundation।

Bangladesh research opportunity:

• Bangla SSL pretrain (BERT-like)।
• Local culture image SSL।
• Medical SSL on local hospital data।
• Foundation model for South Asian context।

মূল উপলব্ধি: Self-supervised pretraining — DL-এর next paradigm। Labeled data dependency drastic reduce। SimCLR, DINO, MAE — different approaches, similar outcome। Foundation model era enable। Bangladesh — local data SSL utilize। Transfer learning evolution — supervised → self-supervised → foundation। Research democratize, deployment simplify।

Bangladesh flora — specific cultural/biological dataset। Practical transfer learning project।

Dataset analysis:

• 1500 image / 15 class = 100 image/class।
• Small but workable।
• Domain — natural image (ImageNet similar)।
• Class imbalance possible (some flower rare)।

Bangladesh flowers special:

• Shapla (water lily) — national flower।
• Kadam, Krishnachura, Bakul — cultural significance।
• Visually similar species — fine-grained challenge।
• Lighting, season variation।

Model selection:

• Tier 1 (default): ResNet-50 pretrained ImageNet।
• Tier 2: EfficientNet-B0 (efficient)।
• Tier 3: DINOv2 (foundation)।
• Tier 4: Plant-specific (PlantNet pretrained)।

Recommended: ResNet-50 + DINOv2:

import torch
import torch.nn as nn
import torchvision.models as models
import torchvision.transforms as T

# ResNet-50 ImageNet
resnet = models.resnet50(weights='IMAGENET1K_V2')
resnet.fc = nn.Sequential(
    nn.Dropout(0.5),
    nn.Linear(2048, 15),
)

# Or DINOv2 (more modern)
# dino = torch.hub.load('facebookresearch/dinov2',
#                       'dinov2_vitb14')

Training pipeline:

train_transform = T.Compose([
    T.Resize(256),
    T.RandomResizedCrop(224, scale=(0.8, 1.0)),
    T.RandomHorizontalFlip(),
    T.RandomRotation(15),
    T.ColorJitter(0.3, 0.3, 0.3, 0.1),
    T.ToTensor(),
    T.Normalize([0.485, 0.456, 0.406],
                [0.229, 0.224, 0.225]),
])

val_transform = T.Compose([
    T.Resize(256),
    T.CenterCrop(224),
    T.ToTensor(),
    T.Normalize([0.485, 0.456, 0.406],
                [0.229, 0.224, 0.225]),
])

Strategy 1: Quick start (feature extract):

for p in resnet.parameters():
    p.requires_grad = False
for p in resnet.fc.parameters():
    p.requires_grad = True

optim = torch.optim.AdamW(
    resnet.fc.parameters(),
    lr=1e-3, weight_decay=0.01)
# 20 epoch — 80% accuracy reasonable

Strategy 2: Full fine-tune:

for p in resnet.parameters():
    p.requires_grad = True

# Discriminative LR
params = [
    {'params': resnet.conv1.parameters(),  'lr': 1e-5},
    {'params': resnet.layer1.parameters(), 'lr': 1e-5},
    {'params': resnet.layer2.parameters(), 'lr': 5e-5},
    {'params': resnet.layer3.parameters(), 'lr': 1e-4},
    {'params': resnet.layer4.parameters(), 'lr': 5e-4},
    {'params': resnet.fc.parameters(),     'lr': 1e-3},
]
optim = torch.optim.AdamW(params, weight_decay=0.01)
sched = torch.optim.lr_scheduler.CosineAnnealingLR(
    optim, T_max=30)
# 30 epoch — 90%+ accuracy

Augmentation specific to flowers:

• Lighting jitter — outdoor variation।
• Rotation — flower pose।
• Color shift — season/time।
• Random crop — different parts focus।

Class imbalance handling:

from torch.utils.data import WeightedRandomSampler

# Compute class frequencies
class_counts = [count_per_class[i] for i in range(15)]
class_weights = 1.0 / torch.tensor(class_counts).float()
sample_weights = class_weights[targets]

sampler = WeightedRandomSampler(
    sample_weights, len(sample_weights))
loader = DataLoader(dataset, batch_size=32,
                    sampler=sampler)

Validation strategy:

• 5-fold cross-validation।
• Stratified split (class balance)।
• Hold-out test — different photographer/location।
• Top-1, top-3 accuracy।

Common errors visualize:

• Confusion matrix।
• Common misclassification pair।
• Hard examples — manual review।
• Domain-specific insight।

Test-time augmentation:

def tta_predict(model, x, n=5):
    model.eval()
    preds = []
    for _ in range(n):
        x_aug = augment(x)
        with torch.no_grad():
            preds.append(F.softmax(model(x_aug), -1))
    return torch.stack(preds).mean(0)

Ensemble:

• 5 fold model average।
• Different architecture combine।
• +2-3% accuracy gain।
• Production deployment cost vs benefit।

Mobile deployment:

• EfficientNet-B0 — small + fast।
• Quantization (INT8)।
• ONNX export।
• Bangla flower app।

Bangladesh-specific deployment:

• Bangla flower name + scientific।
• Cultural significance display।
• Medicinal use info (where applicable)।
• Educational content।

Active learning:

• Uncertain prediction — expert review।
• Dataset grow iteratively।
• Model improvement continuous।

Interpretability:

• Grad-CAM — model attention।
• Feature visualization।
• "Why this flower?" explanation।

Use cases:

• Educational app for school children।
• Botanical research tool।
• Tourism guide।
• Conservation monitoring।

Realistic expectation:

• 1500 image, 15 class — 90-95% achievable।
• Confusion in similar species — manual fallback।
• Production ready with TTA + ensemble।

মূল উপলব্ধি: Bangladesh flower classifier — practical transfer learning project। ResNet-50 ImageNet sufficient starting point। Discriminative LR + augmentation + class balance — recipe। 90%+ accuracy achievable। Mobile deployment quantized model। Educational + cultural app value। Bangladesh-এ AI accessibility — pretrained model leverage essential। Local domain global tool integration।