পাঠ ৩৮ · ৪০-এর মধ্যে · মডিউল ৫
Home / AI Courses / ডিপ লার্নিং / GPU ও CUDA

GPU programming ও CUDA ধারণা

GPU & CUDA basics — parallel compute for DL
৮ মিনিট পড়া মাঝারি · Intermediate PyTorch hands-on

এই পাঠে যা শিখবেন

  • CPU vs GPU — architectural পার্থক্য কেন matter
  • CUDA programming model — kernel, thread, block, grid
  • Memory hierarchy — global, shared, register
  • PyTorch GPU usage — device, transfer, AMP
  • Multi-GPU training — DDP, FSDP
  • Common pitfalls — host-device transfer overhead

১ · GPU কেন এত আলাদা

CPU — few powerful cores (৮-১৬), each handle complex branching, large cache। Optimized for sequential work।

GPU — thousands of simpler cores। Same operation simultaneously on different data। Ideal for matrix multiplication, image processing — DL-এর মূল operation।

একটি ৭৬৮×৭৬৮ matrix multiplication — CPU-তে ~১০০ ms, NVIDIA A100-এ ~০.৫ ms। ২০০x faster। Deep learning সম্ভব হয়েছে এই difference-এর কারণে।

SIMT — Single Instruction, Multiple Threads

GPU-এর core paradigm — একই instruction হাজার thread-এ একসাথে execute, কিন্তু প্রতিটি thread আলাদা data-তে। Image-এর প্রতিটি pixel-এ একই brightness adjust — perfect SIMT match।

২ · CUDA programming model

NVIDIA-এর CUDA (Compute Unified Device Architecture) — GPU-তে general-purpose computing। C-extension।

  • Kernel: GPU-তে চলা function। __global__ keyword।
  • Thread: single execution unit। Each thread একটি element-এ কাজ করে।
  • Block: thread group (e.g., ২৫৬ thread)। Same SM-এ চলে, shared memory access।
  • Grid: all blocks। Total workload।
  • Warp: ৩২ thread একসাথে SIMT — hardware-level।

Thread index dimensions: threadIdx.x, blockIdx.x, blockDim.x। Combined position:

$$\text{global\_id} = \text{blockIdx.x} \times \text{blockDim.x} + \text{threadIdx.x}$$

৩ · Simple CUDA kernel — vector add

CUDA C — vector add kernel 
__global__ void vec_add(float* a, float* b, float* c, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) {
        c[i] = a[i] + b[i];
    }
}

// Launch from host:
int n = 1000000;
int threads = 256;
int blocks  = (n + threads - 1) / threads;
vec_add<<<blocks, threads>>>(d_a, d_b, d_c, n);

১M element add — CPU ~১ ms, GPU ~১০ μs। Massively parallel।

৪ · Memory hierarchy

GPU-এর performance memory access pattern-এর উপর hugely depend।

  • Register (per thread): fastest, ছোট। Local variables।
  • Shared memory (per block): ফাস্ট, ১০০ KB। Block-এর thread-গুলো cooperate।
  • L1/L2 cache: hardware-managed।
  • Global memory: GPU's main DRAM — VRAM, ১০-৮০ GB। Slow।
  • Host memory (CPU RAM): GPU access slow — PCIe transfer।

Coalesced access — adjacent thread adjacent memory address access করলে — single transaction। Misaligned-এ ১০x slow।

GPU memory hierarchy — speed vs capacity Register (fast, tiny) → Shared → L2 → Global (slow, big) Register (per thread) ~256 KB total · 1 cycle Shared memory (per block) ~100 KB · ~5 cycle L1 / L2 cache ~40 MB · ~30 cycle Global memory (VRAM) 10-80 GB · ~500 cycle Host memory (CPU RAM, via PCIe) expensive transfer · avoid in inner loop Fastest Slowest Smallest Biggest
GPU memory hierarchy — register fastest tiny, global slowest big। Optimization-এ shared memory utilization ও coalesced access critical।

৫ · PyTorch — GPU ব্যবহার

Python · PyTorch GPU 
import torch

# Check GPU availability
print(torch.cuda.is_available())          # True if NVIDIA GPU
print(torch.cuda.device_count())          # number of GPUs
print(torch.cuda.get_device_name(0))      # e.g., 'NVIDIA A100'

# Tensor on GPU
x = torch.randn(1000, 1000, device='cuda')
y = torch.randn(1000, 1000).cuda()         # alternative
z = x @ y                                   # GPU matmul

# Bring back to CPU
z_cpu = z.cpu()
print(z_cpu.shape)

# Best practice — single device variable
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = MyModel().to(device)
data  = data.to(device)

    


    
৬ · Mixed precision training (AMP)

    

      FP32 (32-bit float) — default। FP16 (16-bit) — 2x faster, half memory। কিন্তু gradient overflow risk।
    

    

      AMP (Automatic Mixed Precision): forward FP16, optimizer FP32 — best of both। Gradient scaling overflow rod।
    


    

      

        
        Python · AMP training
      

      
import torch
from torch.cuda.amp import autocast, GradScaler

scaler = GradScaler()
model = MyModel().cuda()
optimizer = torch.optim.Adam(model.parameters())

for x, y in loader:
    x, y = x.cuda(), y.cuda()
    optimizer.zero_grad()

    with autocast():                # FP16 forward
        pred = model(x)
        loss = criterion(pred, y)

    scaler.scale(loss).backward()    # scaled gradient
    scaler.step(optimizer)            # unscale + step
    scaler.update()

      

        
        
      

      

    


    

      Modern alternative — BF16 (bfloat16) — Ampere+ GPU-তে। Same range FP32, half memory। No scaling needed।
    


    
৭ · Multi-GPU training

    
    
      
  • DataParallel (DP): simple, single-process। Slow due to GIL।
    • 
      
  • DistributedDataParallel (DDP): recommended। Each GPU separate process। Gradient AllReduce sync।
    • 
      
  • FSDP (Fully Sharded Data Parallel): very large model — parameter shard across GPU। LLaMA 70B-এর জন্য essential।
    • 
      
  • Pipeline parallel: different layer different GPU। Latency hide।
    • 
      
  • Tensor parallel: single layer split। Megatron-LM-এ।
    • 
    


    
৮ · Common pitfalls

    
    
      
  • CPU↔GPU transfer overhead: Inner loop-এ .cpu() avoid।
    • 
      
  • Synchronous timing: torch.cuda.synchronize() before timing — async kernel।
    • 
      
  • Memory leak: training loop-এ retained reference — OOM।
    • 
      
  • Batch size — too small: GPU underutilized। Large batch + gradient accumulation।
    • 
      
  • Mixed-device tensors: "Expected all tensors on same device" — common error।
    • 
      
  • Pinned memory: pin_memory=True in DataLoader — faster H2D transfer।
    • 
    


    
৯ · Hardware landscape

    
    
      
  • NVIDIA H100 (২০২২): ৮০GB HBM, FP16 ~১৯৮৯ TFLOPS — modern LLM training।
    • 
      
  • A100: ৪০/৮০GB, ৩১২ TFLOPS — most cloud GPU।
    • 
      
  • RTX 4090: ২৪GB, consumer ~৮২ TFLOPS — fine-tuning workhorse।
    • 
      
  • AMD MI300, Google TPU: alternative — software stack different।
    • 
      
  • Apple M-series: Metal backend, PyTorch MPS support।
    • 
    


    

      Bangladesh-এ GPU expensive — cloud (AWS, GCP, Lambda Labs) practical। Free Colab/Kaggle GPU starting point। L39-এ এই ভিত্তির উপর — MNIST classifier-এর full project।
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন

    
প্রতিটি প্রশ্ন নিজে কিছুক্ষণ ভাবুন — তারপর "→ উত্তর" চাপুন।


    

      
        প্র ০১
        "GPU faster than CPU" — সব workload-এ সত্যি না। কোথায় CPU বেশি ভালো? কোথায় GPU?
      
      

        

          GPU vs CPU — workload-এর nature-এর উপর depend।
        

        
GPU-favored workload:

        
    
          
  • Matrix multiplication — DL-এর core।
    • 
          
  • Image/video processing — pixel-parallel।
    • 
          
  • Monte Carlo simulation — independent samples।
    • 
          
  • Cryptocurrency mining — hash parallel।
    • 
          
  • Scientific computing — vector operations।
    • 
        

        
CPU-favored workload:

        
    
          
  • Sequential algorithm — no parallelism।
    • 
          
  • Branch-heavy code — divergent control flow।
    • 
          
  • Small workload — kernel launch overhead dominate।
    • 
          
  • Database query — random memory access।
    • 
          
  • Web server — many small request।
    • 
        

        
GPU overhead sources:

        
    
          
  • Kernel launch — ~10 μs overhead।
    • 
          
  • CPU↔GPU memory transfer — PCIe slow।
    • 
          
  • Synchronization — async by default।
    • 
          
  • Initialization cost।
    • 
        

        
When CPU win:

        
    
          
  • Small tensor operation।
    • 
          
  • Sparse data — irregular access।
    • 
          
  • Tree algorithms — hard to parallel।
    • 
          
  • Sequential RNN inference (small batch)।
    • 
        

        
Empirical thresholds:

        
    
          
  • Matrix size > ১০০×১০০ — GPU usually win।
    • 
          
  • Element-wise op on >1M element — GPU।
    • 
          
  • Convolution on >32×32 image — GPU।
    • 
          
  • Below threshold — CPU comparable।
    • 
        

        
Branch divergence:

        
    
          
  • Warp-এর thread-গুলো diverge করলে — serial execution।
    • 
          
  • If-else heavy → GPU efficiency drop।
    • 
          
  • SIMT model penalty।
    • 
        

        
Memory access pattern:

        
    
          
  • Random access — GPU slow।
    • 
          
  • Coalesced access — GPU optimal।
    • 
          
  • Cache hierarchy difference।
    • 
        

        
Modern architecture:

        
    
          
  • CPU SIMD (AVX-512) — narrow gap।
    • 
          
  • Apple M-series — unified memory।
    • 
          
  • NVIDIA Grace Hopper — CPU+GPU integrate।
    • 
          
  • Heterogeneous compute future।
    • 
        

        
DL-specific:

        
    
          
  • Training — GPU dominant।
    • 
          
  • Inference small batch — CPU competitive।
    • 
          
  • Edge deployment — mobile NPU।
    • 
          
  • ONNX runtime — multi-platform।
    • 
        

        
Cost consideration:

        
    
          
  • GPU $/hour high — only when compute bound।
    • 
          
  • CPU sufficient — cheaper for small models।
    • 
          
  • Cloud spot instance — GPU cost dynamic।
    • 
        

        
Bangladesh context:

        
    
          
  • GPU access limited — cloud + Colab।
    • 
          
  • CPU servers cheaper local।
    • 
          
  • Inference often CPU OK।
    • 
          
  • Training cloud-heavy।
    • 
        

        

          মূল উপলব্ধি: GPU = parallel compute hammer। সব problem nail নয়। Sequential, branch-heavy, small-workload — CPU win। DL training mostly GPU। Inference workload-dependent। Hardware choice = problem-aware।
        

      

    


    

      
        প্র ০২
        FP16 mixed precision-এ "gradient scaling" কেন দরকার? FP16 যথেষ্ট না কেন pure?
      
      

        

          Mixed precision — DL training-এর critical optimization। Gradient scaling crucial detail।
        

        
FP16 limitation:

        
    
          
  • Range $\sim 6 \times 10^{-5}$ to $65504$।
    • 
          
  • Small gradient — underflow zero।
    • 
          
  • Precision $\sim 0.1\%$।
    • 
        

        
Gradient distribution:

        
    
          
  • Late training — gradient magnitude small।
    • 
          
  • $10^{-7}$ to $10^{-3}$ common।
    • 
          
  • Below FP16 minimum — flush to zero।
    • 
          
  • Training breaks।
    • 
        

        
Gradient scaling fix:

        
    
          
  • Loss multiply by scale $S$ (e.g., $2^{15}$)।
    • 
          
  • Gradient automatically scaled by $S$।
    • 
          
  • FP16 representable now।
    • 
          
  • Optimizer step — unscale by $S$।
    • 
        

        
Dynamic scaling:

        
    
          
  • Start small scale।
    • 
          
  • Overflow detected — halve scale, skip step।
    • 
          
  • No overflow N steps — double scale।
    • 
          
  • Adaptive optimal।
    • 
        

        
Alternative — BF16:

        
    
          
  • Range same FP32 ($10^{-38}$ to $10^{38}$)।
    • 
          
  • Precision lower (8-bit mantissa)।
    • 
          
  • No scaling needed।
    • 
          
  • Ampere+ GPU support।
    • 
        

        
BF16 vs FP16 trade-off:

        
FP16:

        
    
          
  • Higher precision (10-bit mantissa)।
    • 
          
  • Range issue — scaling needed।
    • 
          
  • Older hardware support।
    • 
        

        
BF16:

        
    
          
  • Better range।
    • 
          
  • Lower precision।
    • 
          
  • Simpler code।
    • 
          
  • Modern preferred।
    • 
        

        
Mixed precision details:

        
    
          
  • Forward FP16/BF16 — fast tensor core।
    • 
          
  • Master weight FP32।
    • 
          
  • Optimizer state FP32।
    • 
          
  • Gradient compute mixed।
    • 
        

        
Memory savings:

        
    
          
  • Activation 2x reduction।
    • 
          
  • Weight 2x (master keep FP32)।
    • 
          
  • Total memory ~50% reduce।
    • 
          
  • Larger batch possible।
    • 
        

        
Speed gains:

        
    
          
  • Tensor core utilization।
    • 
          
  • FP16 ~2x throughput FP32 (V100+)।
    • 
          
  • BF16 same as FP16।
    • 
          
  • Training 1.5-2x faster।
    • 
        

        
Numerical stability concern:

        
    
          
  • BatchNorm স্টatistics — FP32 keep।
    • 
          
  • Loss FP32 computation।
    • 
          
  • Softmax — careful।
    • 
          
  • Edge cases handled by AMP।
    • 
        

        
Modern best practice:

        
    
          
  • BF16 if hardware support।
    • 
          
  • FP16 + AMP otherwise।
    • 
          
  • FP32 master weight always।
    • 
          
  • Gradient checkpointing complement।
    • 
        

        

          মূল উপলব্ধি: FP16 range insufficient — gradient underflow। Scaling — math trick to representable range। BF16 modern alternative — no scaling। Mixed precision — speed + memory + accuracy balance। Production training standard।
        

      

    


    

      
        প্র ০৩
        DDP (DistributedDataParallel) "AllReduce" — কী এবং কেন slow networking-এ DDP scale ভাল না?
      
      

        

          AllReduce — distributed training-এর fundamental operation। Network bottleneck-এর key reason।
        

        
AllReduce mechanics:

        
    
          
  • Each GPU compute gradient।
    • 
          
  • Sum gradient across all GPU।
    • 
          
  • Result distribute back all GPU।
    • 
          
  • Each GPU same gradient।
    • 
          
  • Identical model update।
    • 
        

        
Algorithms:

        
Ring AllReduce:

        
    
          
  • GPU ring topology।
    • 
          
  • Bandwidth-optimal।
    • 
          
  • $2(N-1)/N$ bandwidth utilized।
    • 
          
  • NCCL implementation।
    • 
        

        
Tree AllReduce:

        
    
          
  • Latency-optimal।
    • 
          
  • $\log N$ steps।
    • 
          
  • Smaller data better।
    • 
        

        
Network bottleneck:

        
    
          
  • Gradient size = model size (bytes)।
    • 
          
  • LLaMA-7B: ১৪GB FP16 gradient।
    • 
          
  • Per step communicate this volume।
    • 
          
  • Network bandwidth critical।
    • 
        

        
Bandwidth requirement:

        
    
          
  • NVLink (intra-node): ৬০০ GB/s।
    • 
          
  • InfiniBand: ৫০ GB/s।
    • 
          
  • Ethernet 10G: ১.২৫ GB/s।
    • 
          
  • Ethernet 1G: ০.১২৫ GB/s।
    • 
        

        
Slow network impact:

        
    
          
  • Ethernet 1G — ১৪GB transfer ~১১২ second per step।
    • 
          
  • Compute time only seconds।
    • 
          
  • Network dominate — GPU idle।
    • 
          
  • Effective parallelism কম।
    • 
        

        
Optimization techniques:

        
(১) Gradient compression:

        
    
          
  • Quantize gradient to INT8।
    • 
          
  • 4x bandwidth reduction।
    • 
          
  • Slight accuracy loss।
    • 
        

        
(২) Gradient overlap:

        
    
          
  • Compute backward during AllReduce।
    • 
          
  • Hide communication latency।
    • 
          
  • PyTorch DDP default।
    • 
        

        
(৩) Bucket size tuning:

        
    
          
  • Small bucket — more concurrent communication।
    • 
          
  • Large bucket — less overhead।
    • 
          
  • Trade-off।
    • 
        

        
(৪) Local steps:

        
    
          
  • Multiple steps before sync।
    • 
          
  • Federated learning approach।
    • 
          
  • Asynchronous training।
    • 
          
  • Convergence trade-off।
    • 
        

        
FSDP advantage slow network:

        
    
          
  • Parameter shard — each GPU holds part।
    • 
          
  • Communicate parameter chunk during use।
    • 
          
  • Memory savings significant।
    • 
          
  • Communication amortized।
    • 
        

        
Network topology:

        
    
          
  • Single node multi-GPU — NVLink fast।
    • 
          
  • Multi-node — InfiniBand essential।
    • 
          
  • Cloud — high bandwidth instances।
    • 
          
  • Bangladesh — limited high-bandwidth infrastructure।
    • 
        

        
Bandwidth-aware training:

        
    
          
  • Local development — single GPU।
    • 
          
  • Scale up — same node initially।
    • 
          
  • Multi-node — careful infrastructure।
    • 
          
  • Cloud GPU instance carefully selected।
    • 
        

        
Modern alternatives:

        
    
          
  • Pipeline parallelism — less communication।
    • 
          
  • Tensor parallelism — intra-node।
    • 
          
  • Hybrid — Megatron-LM style।
    • 
          
  • 3D parallelism — DeepSpeed।
    • 
        

        
Bangladesh practical:

        
    
          
  • Single GPU — most common।
    • 
          
  • Multi-GPU rare — research lab।
    • 
          
  • Cloud A100 ($1-2/hour) — short bursts।
    • 
          
  • Pretrained model fine-tune — manageable।
    • 
        

        

          মূল উপলব্ধি: AllReduce — gradient sync mechanism। Bandwidth-bound — slow network problem। Ring algorithm optimal usually। Compression, overlap, FSDP — mitigation। Bandwidth = scale limit। Hardware investment critical for scale।
        

      

    


    

      
        প্র ০৪
        Bangladesh-এর একটি team ML model train করতে চাচ্ছে কিন্তু GPU access সীমিত। কী options + cost-effective strategy?
      
      

        

          Practical Bangladesh ML reality — GPU access constraint। Strategy critical।
        

        
GPU access options:

        
(১) Free tier:

        
    
          
  • Google Colab: Tesla T4, 12hr session।
    • 
          
  • Kaggle: 30hr/week T4/P100।
    • 
          
  • Paperspace: limited free GPU।
    • 
          
  • Hugging Face Spaces: CPU + occasional GPU।
    • 
        

        
(২) Cloud paid:

        
    
          
  • Google Colab Pro: $10/month, A100 access।
    • 
          
  • AWS: spot p3 ~$1-3/hour।
    • 
          
  • GCP: A100 ~$3/hour।
    • 
          
  • Lambda Labs: A100 ~$1.10/hour।
    • 
          
  • Vast.ai: GPU rental marketplace।
    • 
          
  • RunPod: spot GPU cheap।
    • 
        

        
(৩) Local hardware:

        
    
          
  • RTX 4090 — $1500-2000।
    • 
          
  • RTX 3090 — $700-1000।
    • 
          
  • Used Tesla cards — $200-500।
    • 
          
  • Bangladesh import duty consideration।
    • 
        

        
(৪) Academic:

        
    
          
  • BUET, NSU GPU labs।
    • 
          
  • Google Research Cloud credits।
    • 
          
  • Microsoft AI for Good।
    • 
          
  • Apply — academic discount।
    • 
        

        
Cost-effective strategy:

        
Phase 1 — Prototype (Colab free):

        
    
          
  • Architecture design।
    • 
          
  • Small data experiments।
    • 
          
  • Hyperparameter exploration।
    • 
          
  • Bug debugging।
    • 
        

        
Phase 2 — Develop (Colab Pro):

        
    
          
  • Medium scale training।
    • 
          
  • Validation runs।
    • 
          
  • $10/month sustainable।
    • 
        

        
Phase 3 — Production (Cloud spot):

        
    
          
  • Final training runs।
    • 
          
  • A100 Lambda/RunPod ~$1/hour।
    • 
          
  • Burst usage।
    • 
        

        
Phase 4 — Inference (CPU/edge):

        
    
          
  • Quantized model।
    • 
          
  • CPU inference acceptable।
    • 
          
  • Production serving।
    • 
        

        
Compute optimization techniques:

        
    
          
  • Mixed precision — 2x speed।
    • 
          
  • Gradient checkpointing — memory save।
    • 
          
  • LoRA fine-tuning — small parameter।
    • 
          
  • QLoRA — 4-bit quantized।
    • 
          
  • Flash attention — memory efficient।
    • 
        

        
Pretrained model leverage:

        
    
          
  • Start from BERT, LLaMA, ResNet।
    • 
          
  • Fine-tune small data।
    • 
          
  • Avoid from-scratch training।
    • 
          
  • HuggingFace ecosystem।
    • 
        

        
Bangla-specific resources:

        
    
          
  • BanglaBERT pretrained।
    • 
          
  • BUET datasets।
    • 
          
  • Bangla NLP corpus।
    • 
          
  • Google's AI4Bharat।
    • 
        

        
Smart workflow:

        
# Tier-based training
# 1. Local CPU debug
model = MyModel()
loss = train_step(model, sample_batch)

# 2. Colab GPU prototype
# Same code, different runtime
model = model.cuda()
train_full(model, small_data)

# 3. Lambda A100 production
# Distributed if needed
model = model.cuda()
train_full(model, full_data, distributed=True)

        
Inference deployment:

        
    
          
  • ONNX export — universal format।
    • 
          
  • TensorRT optimization।
    • 
          
  • Docker containerize।
    • 
          
  • Cloud GPU API service।
    • 
        

        
Cost monitoring:

        
    
          
  • Set spending alerts।
    • 
          
  • Spot instance for non-critical।
    • 
          
  • Auto-shutdown idle GPU।
    • 
          
  • Profile before scaling।
    • 
        

        
Community resources:

        
    
          
  • Bangladesh AI/ML Facebook groups।
    • 
          
  • Local meetups — knowledge share।
    • 
          
  • BUET, NSU collaborations।
    • 
          
  • Industry-academia bridges।
    • 
        

        
Long-term strategy:

        
    
          
  • Local GPU investment — RTX 3090/4090।
    • 
          
  • Scale to cloud bursts।
    • 
          
  • Pretrained model focus।
    • 
          
  • Domain expertise + AI।
    • 
        

        

          মূল উপলব্ধি: Bangladesh GPU constraint — multi-tier strategy। Free → Colab Pro → Spot cloud → Edge। Pretrained model leverage critical। Optimization (LoRA, quantization) essential। Cost-conscious — production ML viable। Bangladesh AI ecosystem growing।
        

      

    

  


  

    
 অনুশীলন

    
    
      
  1. 
        Benchmark: CPU vs GPU matmul ১০০০×১০০০ — speedup measure (Colab GPU)।
        
    
          
          
    
            
    import torch, time
N = 1000
# CPU
a = torch.randn(N, N); b = torch.randn(N, N)
t0 = time.time(); c = a @ b; t_cpu = time.time() - t0

# GPU
a, b = a.cuda(), b.cuda()
torch.cuda.synchronize()
t0 = time.time(); c = a @ b; torch.cuda.synchronize()
t_gpu = time.time() - t0
print(f"CPU: {t_cpu*1000:.1f} ms")
print(f"GPU: {t_gpu*1000:.1f} ms")
print(f"Speedup: {t_cpu/t_gpu:.1f}x")
    
            
    Typical Colab T4: ~৫০ ms CPU, ~১ ms GPU → ৫০x speedup।
    
          
    
        
    
      
    2. 

      
  2. 
        AMP integration: Existing training loop-এ autocast + GradScaler add।
        
    
          
          
    
            
    from torch.cuda.amp import autocast, GradScaler
scaler = GradScaler()
for x, y in loader:
    x, y = x.cuda(), y.cuda()
    optimizer.zero_grad()
    with autocast():
        pred = model(x)
        loss = criterion(pred, y)
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()
    
          
    
        
    
      
    3. 

      
  3. 
        চিন্তা: Why torch.cuda.synchronize() দরকার GPU benchmarking-এ?
        
    
          
          
    
            
    CUDA kernel launch asynchronous — kernel-launch return immediately, GPU work later। Without sync — timing measure শুধু kernel launch overhead, actual computation নয়।
    
            
    torch.cuda.synchronize() — host wait until all queued GPU work complete। Accurate timing নিশ্চিত। Production training-এ এটা avoid (slow), benchmark-এ দরকার।
    
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন · ABCL TECH-এ আপনার পরবর্তী পদক্ষেপ

    

    

      কোড রানার কাজ না করলে? ব্রাউজারে কাজ না করলে
      Google Colab
      ব্যবহার করুন — Google-এর ফ্রি অনলাইন Python পরিবেশ, শুধু Gmail অ্যাকাউন্ট লাগে।
    

  







  
  
     এই পাঠে ভুল পেয়েছেন? জানান
  





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ৩৭ · Diffusion model