পাঠ ৩০ · ৩২-এর মধ্যে · মডিউল ৪

Home / AI Courses / রিইনফোর্সমেন্ট লার্নিং / প্রজেক্ট: CartPole

প্রজেক্ট: CartPole agent — DQN ও REINFORCE

Project: CartPole agent with DQN & REINFORCE

১৫ মিনিট পড়া hands-on · প্রজেক্ট PyTorch full code

এই পাঠে যা শিখবেন

CartPole environment — state, action, reward structure
DQN পুরো implementation — replay buffer, target net, ε-greedy
REINFORCE পুরো implementation — Monte Carlo policy gradient
দু'টির performance compare ও hyperparameter tuning

১ · CartPole-v1 — environment আগে বুঝি

CartPole — একটি গাড়ি (cart) যা একটি horizontal track-এ left/right যেতে পারে। তার উপরে freely-rotating একটি pole। লক্ষ্য — pole পড়ে যেতে না দেওয়া। প্রতিটি step pole upright থাকলে — reward $+1$।

State (4-D):

$x$: cart position (range ±২.৪, |x|>২.৪ হলে fail)
$\dot{x}$: cart velocity
$\theta$: pole angle (range ±১২° in radians, |θ|>১২° হলে fail)
$\dot{\theta}$: pole angular velocity

Action (discrete, ২): 0 = left push, 1 = right push। Force ±10 N।

Reward: per-step $+1$ যতক্ষণ pole upright। Max ৫০০ steps।

Solved criterion: ১০০ consecutive episodes-এর average return ≥ ৪৭৫ (CartPole-v1)।

কেন CartPole — RL benchmark

ছোট state-space (4-D continuous), tiny action-space (2)। Laptop CPU-তে ৫ মিনিটে solve। Algorithm-এর correctness check-এর প্রথম stop। যদি আপনার DQN/REINFORCE-এ CartPole solve না হয় — code-এ bug, environment-এর সাথে issue না।

২ · Setup ও installation

Bash · Setup

# Python 3.9+
pip install gymnasium torch numpy matplotlib

# Verify CartPole works
python -c "import gymnasium as gym; \
  env = gym.make('CartPole-v1'); \
  obs, _ = env.reset(); \
  print('state shape:', obs.shape, '| actions:', env.action_space.n)"
# state shape: (4,) | actions: 2

৩ · DQN — পুরো implementation

আমরা যে DQN বানাব তাতে থাকবে:

Q-network: 4 → 128 → 128 → 2 (MLP)
Target network: copied every $C=500$ steps
Replay buffer: 50,000 transitions, batch 64
ε-greedy: linear decay 1.0 → 0.05 over ১০K steps
Optimizer: Adam, lr 1e-3
γ = 0.99, Huber loss

Python · PyTorch · DQN full

import gymnasium as gym
import torch, torch.nn as nn
import torch.nn.functional as F
import numpy as np
from collections import deque
import random

# ── Hyperparameters ──────────────────────────────────────────────
GAMMA       = 0.99
LR          = 1e-3
BATCH       = 64
BUFFER_CAP  = 50_000
MIN_BUFFER  = 1_000           # start training only after this many
TARGET_SYNC = 500             # copy online → target every N steps
EPS_START   = 1.0
EPS_END     = 0.05
EPS_DECAY   = 10_000          # linear decay over this many steps
N_EPISODES  = 500
MAX_STEPS   = 500
SEED        = 42

torch.manual_seed(SEED); np.random.seed(SEED); random.seed(SEED)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# ── Q-network ────────────────────────────────────────────────────
class QNet(nn.Module):
    def __init__(self, s_dim=4, a_dim=2):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(s_dim, 128), nn.ReLU(),
            nn.Linear(128, 128),   nn.ReLU(),
            nn.Linear(128, a_dim),
        )
    def forward(self, s):
        return self.net(s)

# ── Replay buffer ────────────────────────────────────────────────
class ReplayBuffer:
    def __init__(self, cap):
        self.buf = deque(maxlen=cap)
    def push(self, s, a, r, s_next, done):
        self.buf.append((s, a, r, s_next, float(done)))
    def sample(self, n):
        batch = random.sample(self.buf, n)
        s, a, r, sn, d = zip(*batch)
        return (torch.tensor(np.array(s),  dtype=torch.float32, device=device),
                torch.tensor(a,            dtype=torch.long,    device=device),
                torch.tensor(r,            dtype=torch.float32, device=device),
                torch.tensor(np.array(sn), dtype=torch.float32, device=device),
                torch.tensor(d,            dtype=torch.float32, device=device))
    def __len__(self):
        return len(self.buf)

# ── Setup ────────────────────────────────────────────────────────
env = gym.make("CartPole-v1")
online = QNet().to(device)
target = QNet().to(device)
target.load_state_dict(online.state_dict())
opt    = torch.optim.Adam(online.parameters(), lr=LR)
buffer = ReplayBuffer(BUFFER_CAP)

def epsilon(step):
    frac = min(1.0, step / EPS_DECAY)
    return EPS_START + frac * (EPS_END - EPS_START)

# ── Training loop ────────────────────────────────────────────────
returns, global_step = [], 0
for episode in range(N_EPISODES):
    s, _ = env.reset(seed=SEED + episode)
    ep_return, done = 0.0, False
    for t in range(MAX_STEPS):
        # ε-greedy action selection
        if random.random() < epsilon(global_step):
            a = env.action_space.sample()
        else:
            with torch.no_grad():
                q = online(torch.tensor(s, dtype=torch.float32, device=device))
                a = int(q.argmax().item())

        s_next, r, term, trunc, _ = env.step(a)
        done = term or trunc
        buffer.push(s, a, r, s_next, term)        # bootstrap on truncation
        s, ep_return = s_next, ep_return + r
        global_step += 1

        # Learning step
        if len(buffer) >= MIN_BUFFER:
            S, A, R, SN, D = buffer.sample(BATCH)
            with torch.no_grad():
                q_next_max = target(SN).max(dim=1).values
                y = R + GAMMA * (1 - D) * q_next_max
            q_pred = online(S).gather(1, A.unsqueeze(1)).squeeze(1)
            loss = F.smooth_l1_loss(q_pred, y)              # Huber
            opt.zero_grad(); loss.backward()
            nn.utils.clip_grad_norm_(online.parameters(), 10.0)
            opt.step()

            if global_step % TARGET_SYNC == 0:
                target.load_state_dict(online.state_dict())

        if done:
            break

    returns.append(ep_return)
    avg100 = np.mean(returns[-100:])
    if (episode + 1) % 10 == 0:
        print(f"Ep {episode+1:3d} | return {ep_return:6.1f} | avg100 {avg100:6.1f} "
              f"| ε {epsilon(global_step):.3f} | buf {len(buffer):5d}")
    if avg100 >= 475 and len(returns) >= 100:
        print(f"\n✓ Solved in {episode+1} episodes! Avg100 = {avg100:.1f}")
        break

env.close()

Typical run-এ episode ~১৫০-২৫০-এ avg100 ৪৭৫ ছাড়িয়ে যায়। CPU-এ ~৫-১০ মিনিট। GPU তেমন benefit নেই (network ছোট, batch ছোট)।

৪ · DQN-এর critical engineering choices

Bootstrap on truncation: CartPole-এ ৫০০ steps-এর পরে truncate (artificial limit)। Truncation ≠ termination — pole still upright, future value আছে। তাই buffer.push(..., term) — শুধু true terminal-এ done flag।
Huber loss > MSE: Q-target outlier-এ less sensitive। Atari থেকে standard practice।
Gradient clipping: Q-target estimate noisy — gradient explosion possible। Clip norm 10।
Target network sync interval: 500 steps — গবেষকরা ১০০-১০০০ recommend। ছোট হলে instability, বড় হলে slow।
ε decay duration: ১০K steps — task-এর scale-এর সাথে। CartPole short, ১০K যথেষ্ট। Atari-তে ১M।

৫ · REINFORCE — পুরো implementation

Policy-based vanilla — Williams (১৯৯২)। Network directly $\pi(a|s)$ output করে softmax-এ। Episode শেষে — discounted return compute, log-prob × return-এর gradient।

Python · PyTorch · REINFORCE full

import gymnasium as gym
import torch, torch.nn as nn
import torch.nn.functional as F
import numpy as np

GAMMA      = 0.99
LR         = 5e-3
N_EPISODES = 1000
SEED       = 42

torch.manual_seed(SEED); np.random.seed(SEED)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

class PolicyNet(nn.Module):
    def __init__(self, s_dim=4, a_dim=2):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(s_dim, 128), nn.ReLU(),
            nn.Linear(128, a_dim),
        )
    def forward(self, s):
        return self.net(s)                          # logits

env    = gym.make("CartPole-v1")
policy = PolicyNet().to(device)
opt    = torch.optim.Adam(policy.parameters(), lr=LR)

def select(s):
    s_t = torch.tensor(s, dtype=torch.float32, device=device)
    logits = policy(s_t)
    dist = torch.distributions.Categorical(logits=logits)
    a = dist.sample()
    return int(a.item()), dist.log_prob(a)

returns_history = []
for episode in range(N_EPISODES):
    s, _ = env.reset(seed=SEED + episode)
    log_probs, rewards = [], []
    done = False
    while not done:
        a, lp = select(s)
        s_next, r, term, trunc, _ = env.step(a)
        log_probs.append(lp); rewards.append(r)
        s = s_next; done = term or trunc

    # Compute discounted returns Gₜ
    G, returns = 0.0, []
    for r in reversed(rewards):
        G = r + GAMMA * G
        returns.insert(0, G)
    returns = torch.tensor(returns, dtype=torch.float32, device=device)
    # Baseline: subtract mean (variance reduction)
    returns = (returns - returns.mean()) / (returns.std() + 1e-8)

    # Policy gradient update
    log_probs = torch.stack(log_probs)
    loss = -(log_probs * returns).sum()
    opt.zero_grad(); loss.backward(); opt.step()

    ep_return = sum(rewards)
    returns_history.append(ep_return)
    avg100 = np.mean(returns_history[-100:])
    if (episode + 1) % 20 == 0:
        print(f"Ep {episode+1:4d} | return {ep_return:6.1f} | avg100 {avg100:6.1f}")
    if avg100 >= 475 and len(returns_history) >= 100:
        print(f"\n✓ Solved in {episode+1} episodes! Avg100 = {avg100:.1f}")
        break

env.close()

REINFORCE high variance — সাধারণত ৩০০-৫০০ episode লাগে। Mean-baseline normalize করায় অনেক stable, যদিও pure variance reduction না (return distribution অনেক skewed)।

৬ · দু'টির comparison

CartPole-এ DQN ~১৫০ episode-এ solve, REINFORCE ~৩৫০। DQN sample-efficient (replay), REINFORCE high-variance কিন্তু simpler।

৭ · Evaluation — trained agent perform check

Python · Evaluation

def evaluate(policy_fn, n_episodes=20, render=False):
    env = gym.make("CartPole-v1", render_mode="human" if render else None)
    returns = []
    for ep in range(n_episodes):
        s, _ = env.reset()
        ep_r, done = 0.0, False
        while not done:
            a = policy_fn(s)
            s, r, term, trunc, _ = env.step(a)
            ep_r += r
            done = term or trunc
        returns.append(ep_r)
    env.close()
    print(f"Avg over {n_episodes}: {np.mean(returns):.1f} ± {np.std(returns):.1f}")
    print(f"Min: {min(returns)}, Max: {max(returns)}")
    return returns

# DQN greedy policy (no ε)
def dqn_greedy(s):
    with torch.no_grad():
        q = online(torch.tensor(s, dtype=torch.float32, device=device))
        return int(q.argmax().item())

# REINFORCE — sample (or use argmax for deterministic)
def reinforce_greedy(s):
    with torch.no_grad():
        logits = policy(torch.tensor(s, dtype=torch.float32, device=device))
        return int(logits.argmax().item())

# Compare
print("DQN evaluation:")
evaluate(dqn_greedy, n_episodes=50)
print("\nREINFORCE evaluation:")
evaluate(reinforce_greedy, n_episodes=50)

৮ · Hyperparameter ablation suggestions

চেষ্টা করে দেখুন:

DQN learning rate: $1\text{e}-2$ (অস্থিতিশীল), $1\text{e}-3$ (default), $1\text{e}-4$ (slow)। Plot training curve।
Buffer size: 5K, 50K, 500K। CartPole ছোট — 50K যথেষ্ট। বড় buffer "stale data"।
Target sync: 100, 500, 5000। ছোট unstable, বড় slow।
REINFORCE GAMMA: 0.95, 0.99, 0.999। 0.99 standard।
Hidden size: 32, 128, 512। CartPole-এ 128 যথেষ্ট, 32 underfit।
Without baseline: REINFORCE-এ (returns - returns.mean()) remove। Variance বিশাল।

Common bugs: (১) gymnasium API new — old env.step(a) ৪-tuple, new ৫-tuple (term, trunc separate)। (২) seed চেক — reproducibility important। (৩) ReplayBuffer.sample not enough buffer — wait until MIN_BUFFER। (৪) REINFORCE return normalization missed — episode-1 return 8, episode-100 return 200 — gradient scale বদলায়।

৯ · Extensions — যেগুলো নিজে try করতে পারেন

Double DQN: $y = r + \gamma Q_{\text{target}}(s', \arg\max_a Q_{\text{online}}(s', a))$ — overestimation bias কমায়।
Dueling DQN: $Q(s,a) = V(s) + A(s,a) - \bar{A}$।
Prioritized Experience Replay: বড় TD error transitions বেশি sample।
Actor-Critic: REINFORCE-এ baseline = learned $V_\phi(s)$।
PPO: CartPole-এ overkill কিন্তু education-এ নিজে implement।
LunarLander-v2: 8-D state, 4 actions। DQN/PPO same code apply।

ভাবনার প্রশ্ন

প্র ০১ কেন CartPole-এ DQN REINFORCE-র চেয়ে সাধারণত সমাধান ৩-৫× দ্রুত করে?

চমৎকার comparison question। মূল কারণগুলো:

(১) Sample reuse: DQN-এর replay buffer-এ প্রতিটি transition বহুবার ব্যবহৃত হয় (প্রায় buffer-size/batch-size = ৭৮০ বার)। REINFORCE প্রতিটি transition একবার দেখে — discard। DQN ৭৮০× sample-efficient।

(২) Per-step learning: DQN প্রতিটি environment step-এ learning step। REINFORCE শুধু episode end-এ। DQN বেশি gradient update পায়।

(৩) Variance: REINFORCE-এর gradient $\sum_t \nabla \log \pi(a_t|s_t) G_t$ — episode return $G_t$ noisy। DQN-এর Q-target episode-independent, lower variance।

(৪) Bootstrapping: DQN $Q(s, a) \leftarrow r + \gamma \max Q(s', a')$ — single-step bootstrap। REINFORCE Monte Carlo (full episode), high variance especially long episodes।

(৫) Off-policy: DQN replay-এ old transitions reuse। REINFORCE on-policy — current policy দিয়ে collect, immediate use।

However — DQN-এর downside:

Bias: Q-overestimation — Double DQN, target net দরকার।
Convergence: non-linear function approximation + bootstrap + off-policy = "deadly triad"। Sometimes diverge।
Discrete only: continuous action handle না — DDPG/SAC লাগে।
Hyperparameter sensitive: target sync, buffer size, ε decay — সব tune।

REINFORCE-এর strengths:

Unbiased: পুরো true policy gradient (কোনো bootstrap bias না)।
Continuous action ready: Gaussian policy easy।
Theoretical foundation: Policy gradient theorem-এর pure form।
Simple: ৩০ lines of code।
PPO ভিত্তি: production-grade RL (PPO, TRPO) REINFORCE-এর descendants।

মূল কথা: CartPole-এ DQN দ্রুত — কিন্তু production RL (continuous control, large-scale) এ PPO/SAC dominant। REINFORCE বুঝে রাখা — pre-requisite।

প্র ০২ "Bootstrap on truncation"-এ কী বললাম এবং এটা miss করলে কী bug?

চমৎকার subtle implementation question।

Setup: CartPole-v1-এ pole পড়ে গেলে — actual terminated=True। কিন্তু ৫০০ steps পেরিয়ে গেলে — truncated=True (artificial limit)। দু'টিই episode end করে, কিন্তু meaning ভিন্ন।

Wrong code:

buffer.push(s, a, r, s_next, done)   # done = term or trunc

Right code:

buffer.push(s, a, r, s_next, term)   # only true termination

কেন matters:

Q-target: $y = r + \gamma (1 - \text{done}) Q(s', \cdot)$।
True terminal: $V(s_{\text{terminal}}) = 0$ — bootstrap multiplier $0$।
Truncation: pole still standing, future value > 0 — bootstrap multiplier $1$।
Wrong code-এ — both treated as terminal। Truncation-এ pole upright হলেও — Q-target $r$ alone, no future।
ফলে Q-network "৫০০ steps-এ everything ends" শেখে — correct strategy underestimate।

Empirical effect:

Without bootstrap-on-truncation: avg100 ~৩৫০-৪০০-এ stuck।
With it: smoothly reaches ৪৭৫+।
Especially noticeable for ১০০+ step episodes।

Generality:

Mountain Car (negative reward each step until goal): truncation-এ bootstrap matters।
Atari (max ১০৮K steps default): rarely truncate, less impact।
Continuing tasks (no terminal): always bootstrap।

Gymnasium API: পুরোনো gym done single boolean। নতুন gymnasium (terminated, truncated) separate — exactly এই issue address করার জন্য।

Best practice:

# PSEUDO
done_bootstrap = terminated      # for Q-target masking
done_episode   = terminated or truncated   # for episode loop end

মূল কথা: RL implementation-এ এই kind subtle correctness issue throughout। "code runs without error" ≠ "code correct"। Reference implementation-এ check, sanity test, debug scripts।

প্র ০৩ আপনি CartPole solve করেছেন। এই code "real-world" robot-এ apply করতে চাইলে কী কী changes? কেন CartPole "toy"?

চমৎকার simulation-to-real question।

CartPole কেন simple:

Tiny state-space (4-D): real robot — joint angles, velocities, vision, force sensors — ১০০-১০০০ dimensions।
Discrete action: real robot — continuous torque/position commands।
No noise: simulator deterministic; real robot — sensor noise, actuator wear, communication delay।
Perfect observation: exact state available; real-world — partial observability, vision-based estimate।
Infinite reset: simulator restart free; real robot — physical reset (humans/robotic arm) costly।
No safety: simulator pole falling free; real robot — dropping → damage।

Required changes:

Continuous action algorithm: DDPG, SAC, PPO instead of DQN।
Larger network: 4-D MLP → CNN for vision, larger hidden।
Domain randomization: simulator-এ vary mass, friction, sensor noise — robust policy।
Sim-to-real transfer: Pretrain in sim, fine-tune real। Or "teacher-student" distillation।
Safety filter: RL action proposed, classical controller (PID) check। Override if dangerous।
Reset automation: auto-reset routine (small robotic motion to recover)।
Sample efficiency: real robot trial expensive — model-based, offline RL, demonstration-based।
Sensor fusion: Kalman filter / learned encoder for noisy observation।
Reward shaping: dense intermediate rewards (joint position, energy use) — sparse "task done" too slow।
Latency handling: action 100ms-old observation-এর basis-এ। Either model latency, or fast sensors।

Real robot benchmark hierarchy:

CartPole-v1 → LunarLander-v2 (still toy)।
Pendulum-v1 (continuous CartPole)।
HalfCheetah / Hopper / Ant (MuJoCo) — physics realistic।
D4RL benchmarks (offline RL)।
Robosuite (manipulation simulator)।
Real robot — huge step। Berkeley, MIT, OpenAI long-running research।

Industrial reality:

Boston Dynamics, Tesla Bot — RL not main। Mostly classical control + ML perception।
OpenAI Rubik's Cube (২০১৯) — full RL, $35M compute, 6 months train। Heavy domain randomization।
Tesla Autopilot — imitation primary, RL minor। Simulator-trained safety-filter overlay।
Boston Dynamics Atlas — model-predictive control (classical), no deep RL।

Bangladesh-specific application:

Agriculture: crop-row navigation tractor — visual servo + simple RL।
Garment factory: precise sewing arm — classical + RL fine-tune।
Delivery drone: route planning + obstacle avoid — hybrid।

মূল কথা: CartPole RL learning-এর classroom। Real-world robot — engineering layered system, RL একটি component। Hype-এর বাইরে — careful, modular, safety-conscious।

প্র ০৪ CartPole "solved" criteria — avg ১০০ episodes ৪৭৫+। কিন্তু "really solved" বুঝতে আর কী এ check করবেন?

চমৎকার evaluation rigor question।

Stated criterion-এর সমস্যা:

Single seed evaluation noisy। Run-to-run variance বিশাল।
Avg over 100 episodes — outlier-dominated।
"Solved at ep 150" misleading — agent stable কি?

Robust evaluation:

Multi-seed (অন্তত ৫): seeds [42, 0, 1, 2, 3]। Mean ± std report। Statistical significance test।
Greedy evaluation: ε=0 (DQN), argmax (REINFORCE)। Stochastic exploration off।
Final performance: last 100 episodes after convergence। Initial training instability ignore।
Robustness to perturbation: small noise on initial state — agent still solve? Generalization indicator।
Sample efficiency: total environment steps to solve, not episodes। CartPole step counts: DQN ~৪০K, REINFORCE ~১০০K।
Compute time: wall-clock to solve।
Visualization: render trained agent — does it look "natural"? Cart oscillates wildly = bad strategy though scoring high।

Production-grade evaluation:

Held-out test envs: slightly different mass, gravity। Generalization check।
Adversarial: small force perturbation per step — "robust" agent recovers।
Comparison baselines: random policy (~22 score), heuristic ("if angle > 0 push right") (~50 score), learned agent (475+)।
Learning curve smoothness: few-spike ramp vs noisy plateau-with-occasional-success। Former more reliable।

Diagnostic plots:

Episode return per episode (raw + smoothed)।
TD loss per training step।
ε decay schedule overlay।
Q-value distribution (DQN) — overestimation check।
Action distribution (REINFORCE) — entropy collapse check।
Gradient norm — exploding/vanishing।

Common false-solved scenarios:

Lucky 100 episodes — next 100 underperform।
Specific seed coincidence।
Hyperparameter sweep cherry-picked।
Code bug ignored truncation → looks solved but isn't।

Repository hygiene:

Random seed fix + report।
Hyperparameters config file।
Tensorboard/W&B logs publish।
Eval script separate — "load checkpoint, run 100 eps, report"।

মূল কথা: RL benchmarking — single number reporting trap। Distribution-aware evaluation, multi-seed, robustness — academic standard। Industry-এ আরও strict — "production CartPole" means deployment-ready, not "passes test"।

অনুশীলন

Run code as-is: উপরের DQN code চালান। Default hyperparameters-এ কত episode-এ solve হলো? Multiple seeds (5টি) চালিয়ে variance report।

Typical: solved in 120-250 episodes across seeds। Variance বড় কারণ replay buffer + ε noise। Mean ~১৭০ episodes, std ~৪০। Bookmark this baseline — অন্য experiments-এ tune compare।
Ablation: target network remove (online network নিজে target হিসেবে)। Training stable থাকে?

Almost always diverge। Q-values explode, agent random। কারণ — target ও prediction same network, "moving target" loop। Target network — DQN-এর critical innovation (DeepMind ২০১৩)। এটাই deadly triad-এর partial solution।
Beyond CartPole: Same DQN code LunarLander-v2-এ apply। কী changes দরকার? Solve হবে কি?

Changes: state dim 4 → 8, action dim 2 → 4। Network input/output dims update। Hyperparameters: longer training (~১০০০ episodes), bigger replay (১০০K), maybe larger network (২৫৬ units)। Solve criterion ২০০। Generally yes — same DQN apply। Atari-তে — frame stacking, CNN, much larger compute।