পাঠ ০৮ · ৪০-এর মধ্যে · মডিউল ১

Home / AI Courses / ডিপ লার্নিং / PyTorch tensor

PyTorch পরিচিতি — tensor

PyTorch intro — tensors as the DL primitive

৭ মিনিট পড়া মাঝারি · Intermediate PyTorch hands-on

এই পাঠে যা শিখবেন

PyTorch tensor কী, NumPy-এর সাথে সম্পর্ক
Tensor creation — সব common উপায়
Indexing, slicing, reshaping
Broadcasting — implicit shape extension
GPU তে tensor — device transfer
Autograd-এর ঝলক — gradient computation

১ · Tensor কী

একটি tensorTensorবহু-মাত্রিক সংখ্যার array — একই dtype-এর। PyTorch-এ DL-এর সব data-র representation। Scalar, vector, matrix — সবই tensor-এর special case। = একটি multi-dimensional array, একই dtype-এর।

0-D (scalar): $5$, $\pi$ — শুধু একটি সংখ্যা।
1-D (vector): $[3, 1, 4]$ — সংখ্যার তালিকা।
2-D (matrix): rows × columns।
3-D: RGB image (height, width, channel)।
4-D: image batch (batch, channel, height, width)।
5-D: video batch (batch, time, channel, height, width)।
আরও বেশি — যেকোনো n-D possible।

Tensor-এর তিন property

১) shape: প্রতিটি dimension-এর size, যেমন $(32, 3, 224, 224)$।
২) dtype: float32, int64, bool, etc।
৩) device: CPU বা GPU।

২ · Tensor Creation

Python · PyTorch

import torch

# Python list থেকে
a = torch.tensor([1, 2, 3])
print(a, a.shape, a.dtype)         # tensor([1, 2, 3]) torch.Size([3]) torch.int64

# Float tensor
b = torch.tensor([1.0, 2.0, 3.0])
print(b.dtype)                      # torch.float32

# 2-D matrix
M = torch.tensor([[1, 2, 3], [4, 5, 6]])
print(M.shape)                      # torch.Size([2, 3])

# Special tensors
z = torch.zeros(3, 4)               # all zeros
o = torch.ones(2, 3)                # all ones
e = torch.eye(3)                    # identity matrix
r = torch.randn(2, 3)               # standard normal
u = torch.rand(2, 3)                # uniform [0, 1)
ar = torch.arange(0, 10, 2)         # [0, 2, 4, 6, 8]
li = torch.linspace(0, 1, 5)        # [0, 0.25, 0.5, 0.75, 1]

print(z); print(r)

৩ · Indexing ও Slicing

Python · PyTorch

import torch

x = torch.tensor([[1, 2, 3, 4],
                  [5, 6, 7, 8],
                  [9, 10, 11, 12]])

print(x[0])         # প্রথম row: [1, 2, 3, 4]
print(x[0, 1])      # row 0, col 1: 2
print(x[:, 0])      # সব row, col 0: [1, 5, 9]
print(x[1:, 1:3])   # row 1 থেকে শেষ, col 1-2: [[6,7],[10,11]]
print(x[-1])        # শেষ row: [9, 10, 11, 12]

# Boolean mask
mask = x > 5
print(x[mask])      # [6, 7, 8, 9, 10, 11, 12]

৪ · Reshaping

Python · PyTorch

import torch

x = torch.arange(12)            # [0, 1, ..., 11], shape (12,)
print(x.shape)

# View — same memory, new shape
y = x.view(3, 4)
print(y); print(y.shape)        # (3, 4)

# Reshape — view-এর safer alternative
z = x.reshape(2, 2, 3)
print(z.shape)                  # (2, 2, 3)

# -1 = "auto-compute"
w = x.view(2, -1)
print(w.shape)                  # (2, 6)

# Squeeze / Unsqueeze — dim add/remove
a = torch.tensor([[1, 2, 3]])   # shape (1, 3)
print(a.squeeze().shape)        # (3,)
print(a.squeeze(0).shape)       # (3,)

b = torch.tensor([1, 2, 3])     # (3,)
print(b.unsqueeze(0).shape)     # (1, 3)
print(b.unsqueeze(1).shape)     # (3, 1)

# Transpose
M = torch.randn(2, 3)
print(M.T.shape)                # (3, 2)

৫ · Operations — Element-wise ও Matrix

Python · PyTorch

import torch

a = torch.tensor([1.0, 2.0, 3.0])
b = torch.tensor([4.0, 5.0, 6.0])

# Element-wise
print(a + b)                # [5, 7, 9]
print(a * b)                # [4, 10, 18]
print(a ** 2)               # [1, 4, 9]
print(torch.exp(a))         # exponential
print(torch.log(a))         # natural log

# Reductions
print(a.sum())              # 6
print(a.mean())             # 2
print(a.max())              # 3
print(a.argmax())           # 2 (index)

# Matrix multiplication
M1 = torch.randn(2, 3)
M2 = torch.randn(3, 4)
print((M1 @ M2).shape)      # (2, 4)
# অথবা torch.matmul(M1, M2)

# Dot product
print(torch.dot(a, b))      # 32 = 1*4 + 2*5 + 3*6

৬ · Broadcasting — Shape Mismatch-এর সমাধান

Broadcasting — দু'টি tensor-এর shape ভিন্ন হলেও — automatic alignment। NumPy-এর মতো।

Python · PyTorch

import torch

# Vector + scalar
a = torch.tensor([1, 2, 3])
print(a + 10)               # [11, 12, 13]

# Matrix + row vector
M = torch.tensor([[1, 2, 3],
                  [4, 5, 6]])         # (2, 3)
v = torch.tensor([10, 20, 30])         # (3,)
print(M + v)
# [[11, 22, 33],
#  [14, 25, 36]]

# Matrix + column vector
c = torch.tensor([[100], [200]])       # (2, 1)
print(M + c)
# [[101, 102, 103],
#  [204, 205, 206]]

Broadcasting rules: trailing dimension থেকে compare। Equal বা একটি ১ হলে compatible। অন্যথায় error। DL-এ — bias addition সব broadcasting।

৭ · GPU তে Tensor

Python · PyTorch

import torch

# GPU available কিনা check
print(torch.cuda.is_available())

# Device-agnostic code
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print("Using device:", device)

# Tensor কে GPU-তে সরানো
x = torch.randn(1000, 1000)
x = x.to(device)
print(x.device)

# সরাসরি GPU-তে create
y = torch.randn(1000, 1000, device=device)

# Op two same-device tensor
z = x + y           # GPU-তে compute

# CPU-তে ফেরত
z_cpu = z.cpu()

সাধারণ ভুল:

Two tensor different device-এ — operation error।
Model GPU-তে, input CPU-তে — এই common mistake।
Solution: X.to(device); model.to(device) — সবসময় same device।

৮ · NumPy ↔ Tensor

Python · PyTorch

import torch
import numpy as np

# NumPy → Tensor
arr = np.array([1.0, 2.0, 3.0])
t = torch.from_numpy(arr)        # shares memory!
print(t)

# Tensor → NumPy
t = torch.tensor([4.0, 5.0, 6.0])
arr = t.numpy()                  # CPU-only
print(arr)

# Memory shared (CPU)
arr[0] = 99
print(t)  # tensor([99., 5., 6.]) — same memory!

৯ · Autograd — Gradient Automatic

PyTorch-এর সবচেয়ে powerful feature। requires_grad=True set করলে — যেকোনো operation গাণিতিকভাবে ট্র্যাক হয় ও gradient compute করা যায়।

Python · PyTorch

import torch

# A simple function: f(x) = x² + 3x + 2
x = torch.tensor(2.0, requires_grad=True)
y = x**2 + 3*x + 2

# Backward — gradient compute
y.backward()

# df/dx = 2x + 3 = 2*2 + 3 = 7
print(x.grad)                    # tensor(7.)

# Multi-variable
a = torch.tensor(3.0, requires_grad=True)
b = torch.tensor(4.0, requires_grad=True)
loss = a**2 + 2*a*b + b**2       # (a+b)²
loss.backward()
print(a.grad)                    # 2a + 2b = 14
print(b.grad)                    # 2a + 2b = 14

Tensor — PyTorch-এর foundation। Shape থেকে DL pipeline-এর প্রতিটি step পর্যন্ত — সব tensor operation।

১০ · একটি Mini-MLP — সব একসাথে

Python · PyTorch

import torch
import torch.nn as nn
import torch.nn.functional as F

# Manual MLP — without nn.Module
torch.manual_seed(0)

# ওজন ও bias
W1 = torch.randn(4, 2, requires_grad=True) * 0.1
b1 = torch.zeros(4, requires_grad=True)
W2 = torch.randn(1, 4, requires_grad=True) * 0.1
b2 = torch.zeros(1, requires_grad=True)

# XOR data
X = torch.tensor([[0.,0.],[0.,1.],[1.,0.],[1.,1.]])
y = torch.tensor([[0.],[1.],[1.],[0.]])

# Train
for epoch in range(2000):
    # Forward
    z1 = X @ W1.T + b1
    a1 = F.relu(z1)
    z2 = a1 @ W2.T + b2

    # Loss (BCE with logits)
    loss = F.binary_cross_entropy_with_logits(z2, y)

    # Backward
    loss.backward()

    # Manual SGD
    with torch.no_grad():
        for p in [W1, b1, W2, b2]:
            p -= 0.1 * p.grad
            p.grad.zero_()

    if epoch % 400 == 0:
        print(f"epoch {epoch}: loss = {loss.item():.4f}")

# Test
with torch.no_grad():
    z1 = X @ W1.T + b1; a1 = F.relu(z1); z2 = a1 @ W2.T + b2
    print("\nFinal:", torch.sigmoid(z2).flatten().tolist())

Tensor + autograd + manual SGD — পুরো DL training pipeline-এর core mechanic। nn.Module, nn.Sequential, optim.SGD — এই লাইনগুলো convenience wrapper।

Tensor PyTorch-এর foundation। Shape mismatch, device mismatch, broadcasting bug — beginner-দের ৭০% issue। প্রতিটি tensor-এ — shape print, device check, dtype confirm — habit তৈরি করুন।

ভাবনার প্রশ্ন

প্রতিটি প্রশ্ন নিজে কিছুক্ষণ ভাবুন — তারপর "→ উত্তর" চাপুন।

প্র ০১ PyTorch tensor আর NumPy ndarray প্রায় একই দেখায়। তবু DL-এ NumPy-র বদলে PyTorch কেন? দু'টোর key পার্থক্য কী?

ML stack-এর design choice-এর ভেতরে যাওয়া।

NumPy strengths:

Mature, ১৯৯৫ থেকে।
Scientific Python ecosystem-এর hub।
Fast CPU computation।
SciPy, scikit-learn, pandas-এর সাথে integration।

PyTorch tensor-এর extra:

GPU support: .to('cuda') — ১০০x speedup। NumPy CPU-only।
Autograd: automatic differentiation। NumPy-তে gradient manually।
Mixed precision: FP16/BF16 native support।
Distributed training: built-in primitives।
Compilation: torch.compile — graph optimization।

API similarities:

Many functions same name: .sum(), .mean(), .reshape()।
Broadcasting rules same।
Indexing similar।
Easy migration।

API differences:

NumPy: arr.reshape(...)। PyTorch: .view() বা .reshape()।
NumPy: np.dot(a, b)। PyTorch: a @ b বা torch.matmul।
NumPy: arr.T। PyTorch: same।
PyTorch-এ requires_grad, device — extra।

Memory sharing:

torch.from_numpy(arr) — same memory (CPU)।
tensor.numpy() — same memory।
One modify-এ other-ও affect।
GPU tensor → numpy = explicit copy।

কখন NumPy-ই যথেষ্ট:

Small data, no DL।
Statistical analysis।
Pre-DL ML (scikit-learn)।
Visualization, plotting।

কখন PyTorch দরকার:

DL — neural network training।
GPU acceleration।
Large-scale matrix operation।
Gradient-based optimization।

Other DL frameworks:

TensorFlow: Google, production-focused।
JAX: Google research, functional, fast।
MXNet, Flax: alternatives।
PyTorch — research dominant, production growing।

Bangladesh adoption:

BUET, IUT, NSU course — PyTorch primary।
Bangla NLP research — PyTorch dominant।
Industry — TensorFlow legacy, PyTorch new projects।

মূল উপলব্ধি: NumPy = scientific Python। PyTorch = NumPy + GPU + autograd। DL-এর জন্য PyTorch indispensable। কিন্তু দু'টো competing না — complementary।

প্র ০২ "Broadcasting" একটা powerful concept — কিন্তু সবচেয়ে বেশি bug-এর source। কোন-কোন subtle case-এ broadcasting silently ভুল করে?

PyTorch debugging-এর সবচেয়ে frustrating শ্রেণী।

Broadcasting rules recap:

Trailing dimension থেকে align।
Equal বা একটি ১ — compatible।
একটি missing — implicit ১।
অন্যথায় shape mismatch error।

Subtle bug (১) — Wrong dimension addition:

a = torch.randn(10, 5)      # (10, 5)
b = torch.randn(10)         # (10,)
c = a + b                   # (10, 5) — broadcasts on dim 1
# কিন্তু intended: per-row addition!
# Solution: b.unsqueeze(1) → (10, 1)

Subtle bug (২) — Loss shape mismatch:

pred = model(X)             # (32, 1)
y = torch.tensor([...])     # (32,)
loss = F.mse_loss(pred, y)  # silent broadcast — WRONG!
# Pred (32, 1) vs y (32,) → broadcasts to (32, 32)
# Solution: pred.squeeze() বা y.unsqueeze(1)

Subtle bug (৩) — Pairwise distance:

X = torch.randn(100, 64)    # 100 vectors, 64-D
# Pairwise distance:
diff = X.unsqueeze(0) - X.unsqueeze(1)
# (1, 100, 64) - (100, 1, 64) = (100, 100, 64) ✓
dist = (diff ** 2).sum(dim=-1).sqrt()   # (100, 100)

Subtle: dimension order important। Wrong unsqueeze position → wrong result।

Subtle bug (৪) — Implicit type promotion:

a = torch.tensor([1, 2, 3])         # int64
b = torch.tensor([1.0, 2.0, 3.0])   # float32
c = a + b                            # float32 (promoted)
# কখনো — int operations expected, float আসে

Subtle bug (৫) — Channel dimension confusion:

# PyTorch: NCHW (batch, channels, H, W)
# TensorFlow: NHWC
# CV libraries (PIL, OpenCV): HWC
img = torch.randn(224, 224, 3)       # HWC
img.permute(2, 0, 1)                  # CHW for PyTorch

Bug (৬) — In-place operations + autograd:

x = torch.tensor(2.0, requires_grad=True)
y = x ** 2
y += 1   # in-place — autograd error sometimes!
# Solution: y = y + 1 (out-of-place)

Bug (৭) — Index from CPU on GPU tensor:

x = torch.randn(10, device='cuda')
idx = [0, 2, 4]              # Python list
x[idx]                        # OK — auto-converts
idx_t = torch.tensor([0, 2, 4])  # CPU tensor
x[idx_t]                      # ERROR — different device

Defensive practices:

Print shapes: debugging mode-এ everywhere।
Assert shapes: production code-এ critical points-এ।
Use einops: rearrange/reduce — explicit dimension naming।
Type check: x.dtype, x.device।
Test with small batch: tensor flow verify।

Tools:

einops: rearrange(x, 'b c h w -> b (h w) c') — readable।
torchtyping: shape annotations।
jaxtyping: type-checked tensors।

মূল উপলব্ধি: Broadcasting — magical when right, evil when wrong। Silent failure-এর master। Defensive programming — DL coding-এর core skill। Print, assert, test — three pillars।

প্র ০৩ GPU-তে computation ১০০x faster — এই common claim কতটা সঠিক? কোন situation-এ GPU benefit কম, কোথায় bottleneck আসে?

ML engineering-এর সবচেয়ে practical performance question।

"১০০x faster" — অর্ধসত্য:

Specific operations (large matrix multiply) — yes, 50-100x।
Average DL workload — 5-30x।
Small workload — sometimes slower (transfer overhead)।

GPU-এর strengths:

হাজার core (CUDA core) — massive parallelism।
Memory bandwidth ১০x CPU।
Specialized matrix multiply (Tensor Cores)।
Half/mixed precision native।

কখন GPU benefit বেশি:

Large batch size — utilization high।
Large matrix multiply (>1024x1024)।
Convolution operations।
Same operation many times (training loop)।

কখন GPU benefit কম:

Small batch: CPU competitive।
Sequential operations: RNN — limited parallelism।
Data preprocessing: often CPU work।
Single inference: latency-bound।

Common bottlenecks:

(১) Data loading: disk → CPU → GPU। GPU idle while waiting।
- Solution: num_workers in DataLoader।
- Pre-fetch with pin_memory=True।
- Cache on SSD।
(২) CPU-GPU transfer: PCIe bottleneck।
- Minimize .to(device) calls।
- Keep data on GPU as long as possible।
- pin_memory for fast transfer।
(৩) Small operations: kernel launch overhead।
- Solution: torch.compile — fuse kernels।
- Batch operations together।
(৪) Memory bandwidth: not compute-bound।
- Use lower precision (FP16/BF16)।
- Gradient checkpointing for memory saving।

Profiling:

torch.profiler — CPU/GPU time breakdown।
nvidia-smi — GPU utilization, memory।
NSight Compute — kernel-level profiling।

GPU utilization rules:

>৯০% utilization — optimal।
৫০-৮০% — improvement possible।
<৫০% — likely data loading bottleneck।

Modern hardware:

NVIDIA H100 — ~3 TFLOPS FP64, 60+ TFLOPS FP16।
A100 — previous gen, still mainstream।
Consumer (RTX 4090) — research, small training।
TPU (Google) — for TF/JAX।

Bangladesh practical:

Cloud GPU rental (Google Colab Pro, Paperspace) — accessible।
Local GPU — RTX 3060/4060, ১২-২৪GB VRAM minimum।
Multi-GPU — research lab।
CPU-only — small models, prototyping।

Cost-aware decisions:

Inference: smaller GPU বা CPU যথেষ্ট।
Training: large GPU।
Fine-tuning: mid-range।
LLM inference: GPU memory critical (24GB+)।

মূল উপলব্ধি: "GPU = fast" — naive। GPU efficient when properly utilized। Profiling, batching, transfer minimization — engineering skills। Naive port to GPU — sometimes slower।

প্র ০৪ Autograd-এর "magic" কীভাবে কাজ করে? backward() call করলে gradient কীভাবে compute হয় ভেতরে?

PyTorch-এর সবচেয়ে elegant engineering — যা DL-কে democratize করেছে।

Autograd-এর core idea:

প্রতিটি tensor operation — graph-এ recorded।
Forward pass build a "computation graph"।
backward() — graph traverse করে chain rule apply।
Result — leaf tensor-এর .grad।

Graph structure:

Node — tensor (with requires_grad)।
Edge — operation।
Each operation node — local gradient store করে।
Leaves — input tensors।
Root — output (loss)।

Forward pass example:

x = torch.tensor(2.0, requires_grad=True)
a = x ** 2          # node: Pow, grad_fn=PowBackward
b = a + 3           # node: Add, grad_fn=AddBackward
y = b * 5           # node: Mul, grad_fn=MulBackward

Each node has grad_fn — backward function।

Backward pass mechanics:

y.backward() — start with $\frac{dy}{dy} = 1$।
Mul node: $\frac{dy}{db} = 5$।
Add node: $\frac{db}{da} = 1$ → $\frac{dy}{da} = 5 \cdot 1 = 5$।
Pow node: $\frac{da}{dx} = 2x = 4$ → $\frac{dy}{dx} = 5 \cdot 4 = 20$।
Result: x.grad = 20।

Chain rule in matrix form:

Vector-Jacobian product (VJP)।
Reverse-mode automatic differentiation।
Memory $O(\text{forward activations})$।
Compute roughly $2 \times$ forward।

Dynamic graph (PyTorch):

Graph build during forward। Run-time।
Different graph each iteration possible।
Pythonic — control flow natural।
vs TF 1.x static graph — define once, run many।

Backward function implementation:

Each operation has C++ implementation।
Forward saves needed values (e.g., input for ReLU)।
Backward uses saved + incoming gradient।
Custom op — torch.autograd.Function।

Memory cost:

All intermediate activations saved।
Deep network — O(layers) memory।
Solution: torch.utils.checkpoint — recompute trade-off।

Detach/no_grad:

x.detach() — break graph, no gradient flow।
torch.no_grad() — context, no graph build।
Inference / frozen layers।

Gradient accumulation:

x.grad accumulates by default!
optimizer.zero_grad()  # always before backward()

Higher-order gradients:

create_graph=True — gradient itself differentiable।
Meta-learning, second-order optimization।
MAML, Hessian-vector products।

Modern alternatives:

torch.func (functorch): functional autograd (jvp, vjp, jacobian)।
JAX: functional, transformations composable।
torch.compile: graph capture + optimization।

মূল উপলব্ধি: Autograd = chain rule automated + dynamic graph + reverse-mode efficient। DL-এর "magic" = engineering elegance। Without autograd — manual gradient = nightmare। Most DL frameworks built on this primitive।

অনুশীলন

Shape: একটি batch of 64 RGB images, 224×224 — PyTorch tensor-এর shape কী?

$(64, 3, 224, 224)$ — NCHW format। Channel = 3 (RGB)।

Code: $f(x, y) = x^2 y + y^3$, $x = 2, y = 3$-এ partial derivatives autograd-এ:

x = torch.tensor(2.0, requires_grad=True)
y = torch.tensor(3.0, requires_grad=True)
f = x**2 * y + y**3
f.backward()
print(x.grad)   # 2xy = 12
print(y.grad)   # x² + 3y² = 4 + 27 = 31

Debug: এই code-এ কী bug:

X = torch.randn(100, 10).cuda()
W = torch.randn(10, 5)
Y = X @ W

Device mismatch — X GPU-তে, W CPU-তে। RuntimeError।

Fix: W = torch.randn(10, 5).cuda() বা both on same device:

device = 'cuda'
X = torch.randn(100, 10, device=device)
W = torch.randn(10, 5, device=device)
Y = X @ W

এই পাঠে যা শিখবেন

১ · Tensor কী

২ · Tensor Creation

৩ · Indexing ও Slicing

৪ · Reshaping

৫ · Operations — Element-wise ও Matrix

৬ · Broadcasting — Shape Mismatch-এর সমাধান

৭ · GPU তে Tensor

৮ · NumPy ↔ Tensor

৯ · Autograd — Gradient Automatic

১০ · একটি Mini-MLP — সব একসাথে

ভাবনার প্রশ্ন

অনুশীলন

আরও পড়ুন · ABCL TECH-এ আপনার পরবর্তী পদক্ষেপ