পাঠ ৩১ · ৩২-এর মধ্যে · মডিউল ৪

Home / AI Courses / রিইনফোর্সমেন্ট লার্নিং / প্রজেক্ট: Atari DQN

প্রজেক্ট: Atari Breakout-এ DQN

Project: DQN on Atari Breakout

১৫ মিনিট পড়া hands-on · প্রজেক্ট CNN + frame stack

এই পাঠে যা শিখবেন

Atari environment preprocessing — grayscale, resize, frame stacking
CNN-based DQN architecture (Mnih ২০১৫ "Nature DQN")
Production-grade replay buffer ও training loop
Hyperparameters ও common pitfalls — Bangladesh-এর কম GPU-তে কীভাবে train

১ · কেন Atari hard?

CartPole 4-D state, ২ actions, ১ minute training. Atari Breakout — একটি 84×84 grayscale frame ৭০৫৬-D, ৪ actions (left/right/fire/no-op), ১০-৫০ million frames train. Difference quantitative + qualitative.

মূল challenges:

High-D observation: raw pixels — features manually extract করা impossible. CNN দরকার।
Partial observability: single frame থেকে ball-এর direction বোঝা না — past frames দরকার।
Sparse reward: brick break-এ +1, এর মাঝে ০। Long horizon credit assignment.
Long horizon: episode ১০K+ frames possible. DQN target stable রাখা কঠিন।
Compute: single training run ১-৫ days A100 GPU. Bangladesh-এ rare resource.

DeepMind Nature DQN (২০১৫)

Mnih et al. — same DQN code, ৪৯ Atari games-এ apply. ২৯টি গেমে human-level+, কিছুতে superhuman. শিরোনাম: "Human-level control through deep reinforcement learning" — RL-এর modern era-র শুরু।

২ · Preprocessing pipeline

Raw Atari frame: 210×160×3 (RGB), 60 FPS. DQN-এ যা যা করা হয়:

Grayscale: 3 channels → 1. Color irrelevant Breakout-এ।
Resize: 210×160 → 84×84. Compute reduce, খেলায় detail enough.
Frame skip: agent প্রতি 4 frames-এ একবার action. Same action 4 frames repeat. Speed 4× + temporal coherence.
Max-pooling 2 consecutive frames: Atari sprite flicker — alternate frames-এ object alternate appear. Pixel-wise max ensure visibility.
Frame stack: last 4 (preprocessed) frames → 4×84×84 tensor. Velocity/direction inferable.
Reward clip: $r \in \{-1, 0, +1\}$। Different games-এ scale uniform. (অর্থ হারায় কিন্তু stability জেতে।)
Episode life: Atari-র অনেক game-এ ৩-৫ "lives". Treat each life-loss as terminal training-এ — বেশি reward signal.

৩ · Setup

Bash · Setup

# Atari ROMs (legal redistribution via ale-py)
pip install gymnasium[atari,accept-rom-license]
pip install torch numpy opencv-python

python -c "
import gymnasium as gym
env = gym.make('ALE/Breakout-v5')
print('actions:', env.action_space.n)
print('obs:', env.observation_space.shape)
"
# actions: 4
# obs: (210, 160, 3)

৪ · Atari wrappers — preprocessing apply

Python · Atari wrappers

import gymnasium as gym
from gymnasium.wrappers import (
    AtariPreprocessing,
    FrameStack,
)

def make_atari(env_id="ALE/Breakout-v5", seed=42):
    env = gym.make(env_id, frameskip=1, repeat_action_probability=0.0,
                   full_action_space=False)
    # AtariPreprocessing handles grayscale, 84x84 resize,
    # frame skip (4), no-op start, max-pool, terminal-on-life-loss.
    env = AtariPreprocessing(
        env,
        noop_max=30,
        frame_skip=4,
        screen_size=84,
        terminal_on_life_loss=True,
        grayscale_obs=True,
        scale_obs=False,
    )
    env = FrameStack(env, num_stack=4)
    env.action_space.seed(seed)
    return env

env = make_atari()
obs, _ = env.reset(seed=42)
print("Stacked obs:", obs.shape)            # (4, 84, 84)
print("Action space:", env.action_space)    # Discrete(4)

Gymnasium-এর built-in wrappers সব Atari preprocessing handle করে — DeepMind-এর exact recipe. আপনি hand-roll-ও করতে পারেন (cv2.resize, cv2.cvtColor) কিন্তু wrappers reliable.

৫ · CNN Q-network

Mnih ২০১৫ Nature architecture:

Python · PyTorch · Nature DQN CNN

import torch, torch.nn as nn
import torch.nn.functional as F

class NatureDQN(nn.Module):
    """DeepMind ২০১৫ Nature paper architecture."""
    def __init__(self, n_actions=4):
        super().__init__()
        self.conv = nn.Sequential(
            nn.Conv2d(4, 32, kernel_size=8, stride=4),  # 4×84×84 → 32×20×20
            nn.ReLU(),
            nn.Conv2d(32, 64, kernel_size=4, stride=2), # 32×20×20 → 64×9×9
            nn.ReLU(),
            nn.Conv2d(64, 64, kernel_size=3, stride=1), # 64×9×9 → 64×7×7
            nn.ReLU(),
        )
        self.fc = nn.Sequential(
            nn.Linear(64 * 7 * 7, 512),
            nn.ReLU(),
            nn.Linear(512, n_actions),
        )

    def forward(self, x):
        # x: (B, 4, 84, 84) uint8 or float
        if x.dtype == torch.uint8:
            x = x.float() / 255.0
        h = self.conv(x)
        h = h.flatten(1)                                 # (B, 64*7*7)
        return self.fc(h)

# Sanity check
net = NatureDQN(n_actions=4)
sample = torch.zeros(1, 4, 84, 84, dtype=torch.uint8)
print("Output Q-values:", net(sample).shape)            # (1, 4)
print("Params:", sum(p.numel() for p in net.parameters())) # ~1.7M

৬ · Memory-efficient replay buffer

Atari replay buffer 1M frames. Naive পাইথন list — 1M × 4 × 84 × 84 × 4 bytes = ~১১৩ GB! সমাধান: uint8 storage + frame deduplication (সব frame stack ছাড়া single frames store).

Python · Memory-efficient replay (single-frame store)

import numpy as np
import torch

class FrameReplay:
    """Stores SINGLE frames (uint8); reconstructs 4-frame stacks on sample."""
    def __init__(self, capacity=1_000_000, frame_dim=(84, 84), n_stack=4):
        self.capacity = capacity
        self.n_stack  = n_stack
        self.frames   = np.zeros((capacity, *frame_dim), dtype=np.uint8)
        self.actions  = np.zeros(capacity, dtype=np.int64)
        self.rewards  = np.zeros(capacity, dtype=np.float32)
        self.dones    = np.zeros(capacity, dtype=np.bool_)
        self.idx      = 0
        self.full     = False

    def push(self, frame, action, reward, done):
        self.frames[self.idx]  = frame
        self.actions[self.idx] = action
        self.rewards[self.idx] = reward
        self.dones[self.idx]   = done
        self.idx = (self.idx + 1) % self.capacity
        if self.idx == 0:
            self.full = True

    def __len__(self):
        return self.capacity if self.full else self.idx

    def _stack(self, idx):
        """Build 4-frame stack ending at idx."""
        frames = []
        for k in range(self.n_stack - 1, -1, -1):
            j = (idx - k) % self.capacity
            frames.append(self.frames[j])
            # If episode boundary inside the window, pad with the latest frame
            if k > 0 and self.dones[(j - 1) % self.capacity]:
                frames = [self.frames[idx]] * (k) + frames
                break
        return np.stack(frames, axis=0)

    def sample(self, batch_size, device):
        max_idx = len(self) - 1
        idx = np.random.randint(self.n_stack, max_idx, size=batch_size)
        # Avoid sampling crossing buffer wrap
        idx = idx[idx != self.idx - 1]

        s   = np.stack([self._stack(i)         for i in idx])
        s_n = np.stack([self._stack((i + 1) % self.capacity) for i in idx])
        a   = self.actions[idx]
        r   = self.rewards[idx]
        d   = self.dones[idx].astype(np.float32)

        return (torch.from_numpy(s).to(device),
                torch.from_numpy(a).to(device),
                torch.from_numpy(r).to(device),
                torch.from_numpy(s_n).to(device),
                torch.from_numpy(d).to(device))

Memory: 1M × 84 × 84 × 1 byte = ~৭ GB. Frame stack runtime-এ build হয় — slower per-sample, কিন্তু RAM accommodatable. Production replay (Dopamine, ALE) এই pattern ব্যবহার করে।

৭ · Training loop — Atari DQN

Python · PyTorch · Atari DQN training

import torch, torch.nn as nn
import torch.nn.functional as F
import numpy as np
import random
from collections import deque

# ── Hyperparameters (DeepMind Nature settings) ─────────────────────
GAMMA          = 0.99
LR             = 2.5e-4
BATCH          = 32
BUFFER_CAP     = 1_000_000
MIN_BUFFER     = 50_000
TARGET_SYNC    = 10_000
EPS_START      = 1.0
EPS_END        = 0.1
EPS_DECAY      = 1_000_000        # frames
TRAIN_FREQ     = 4                # learn every 4 environment steps
TOTAL_FRAMES   = 10_000_000       # 10M frames typical
SEED           = 42

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
torch.manual_seed(SEED); np.random.seed(SEED); random.seed(SEED)

env       = make_atari("ALE/Breakout-v5", seed=SEED)
n_actions = env.action_space.n

online = NatureDQN(n_actions).to(device)
target = NatureDQN(n_actions).to(device)
target.load_state_dict(online.state_dict())
opt    = torch.optim.Adam(online.parameters(), lr=LR, eps=1.5e-4)
buffer = FrameReplay(BUFFER_CAP)

def epsilon(frame):
    f = min(1.0, frame / EPS_DECAY)
    return EPS_START + f * (EPS_END - EPS_START)

obs, _ = env.reset(seed=SEED)
obs = np.asarray(obs)                      # (4, 84, 84) — frame stack already
ep_return, ep_len, returns_log = 0.0, 0, deque(maxlen=100)

for frame in range(1, TOTAL_FRAMES + 1):
    eps = epsilon(frame)
    # ε-greedy on the current 4-frame stack
    if random.random() < eps:
        a = env.action_space.sample()
    else:
        with torch.no_grad():
            q = online(torch.from_numpy(np.asarray(obs)).unsqueeze(0).to(device))
            a = int(q.argmax(1).item())

    next_obs, r, term, trunc, _ = env.step(a)
    next_obs = np.asarray(next_obs)
    done = term or trunc

    # Store ONLY the latest single frame (last channel of stack)
    buffer.push(next_obs[-1], a, np.clip(r, -1, 1), term)
    obs = next_obs
    ep_return += r; ep_len += 1

    # Train step
    if len(buffer) >= MIN_BUFFER and frame % TRAIN_FREQ == 0:
        S, A, R, SN, D = buffer.sample(BATCH, device)
        with torch.no_grad():
            q_next = target(SN).max(1).values
            y = R + GAMMA * (1 - D) * q_next
        q_pred = online(S).gather(1, A.unsqueeze(1)).squeeze(1)
        loss = F.smooth_l1_loss(q_pred, y)
        opt.zero_grad(); loss.backward()
        nn.utils.clip_grad_norm_(online.parameters(), 10.0)
        opt.step()

    if frame % TARGET_SYNC == 0:
        target.load_state_dict(online.state_dict())

    if done:
        returns_log.append(ep_return)
        if len(returns_log) % 10 == 0:
            avg = np.mean(returns_log)
            print(f"frame {frame:9,d} | ep len {ep_len:4d} | ret {ep_return:5.0f} | "
                  f"avg100 {avg:6.2f} | ε {eps:.3f}")
        ep_return, ep_len = 0.0, 0
        obs, _ = env.reset()
        obs = np.asarray(obs)

env.close()

৮ · Expected training trajectory

Atari Breakout DQN — frames vs episode return. ১M frames-এ replay fill, ১০M-এ human-level, ৫০M-এ superhuman, ১৫০M+-এ "wall trick" emerge.

৯ · Bangladesh-এ Atari train — practical tips

আপনার GPU access সীমিত? এই tricks:

Smaller frame budget: Mnih ২০১৫ ২০০M, কিন্তু ১০-২০M-ই পরিচিত performance. Time/cost সাশ্রয়।
Cloud spot instances: Lambda Labs (A100 ~$1.5/hr), RunPod, Vast.ai. ১০M frames ৪-৬ ঘণ্টা।
Google Colab Pro+: A100 access. 1-day continuous train possible.
EnvPool: parallel environment library — same compute-এ ৪-৮× faster.
Dopamine framework: DeepMind-এর reference implementation. Don't reinvent wheel.
CleanRL: Single-file implementations. ছাত্রদের জন্য study করতে easy.
Smaller game: Breakout heavy, Pong/Space Invaders more forgiving.

Common bugs: (১) frame stacking — দেখুন আসলে 4 frames আসছে কি। (২) reward clipping ভুলে গেলে — Q-values explode. (৩) terminal-on-life-loss না করলে — সাকসেস signal স্পার্স। (৪) Adam epsilon ১.৫e-৪ specific — default ১e-৮ unstable atari-তে।

১০ · Beyond Nature DQN — modern variants

আপনার Atari DQN কাজ করছে — পরের steps:

Double DQN (van Hasselt ২০১৬): Q-overestimation কমায়। ১ লাইন code change.
Dueling DQN (Wang ২০১৬): $Q = V + A$ decomposition.
Prioritized Experience Replay (Schaul ২০১৬): বড় TD error transitions বেশি sample.
Rainbow DQN (Hessel ২০১৭): ৬টি improvement একসাথে। SOTA ২০১৭।
R2D2 (২০১৯): recurrent + distributed. Long-horizon.
Agent57 (২০২০): ৫৭ Atari games-এ human-level পার!
MuZero (২০২০): model-based + tree search. Much higher performance.
EfficientZero (২০২১): Atari-100K — ১০০K frames-এ human-level.

ভাবনার প্রশ্ন

প্র ০১ Frame stacking কেন critical? Single frame-এ এমন কী missing?

চমৎকার partial-observability question.

Setup: Atari frame static image. Ball position দেখা যায় — কিন্তু ball-এর velocity, direction? নেই।

Markov property violation: RL theory ধরে নেয় current state $s_t$ থেকে future predictable. Atari frame markov না — past frames-এ velocity হিডেন।

Frame stack solution: last 4 frames concat — ball position 4 timesteps-এর। Difference থেকে velocity, acceleration inferable. Network implicit shape derive.

Why 4 frames specifically:

2 frames — position + velocity.
3 frames — + acceleration.
4 frames — + jerk (rate of change of acceleration).
Empirically 4 sweet spot. 8 marginal gain, 2× memory.

Alternative — RNN/LSTM:

Recurrent DQN (DRQN) — frame stack-এর বদলে hidden state propagate.
Pros: arbitrary history, no fixed window.
Cons: training slower, BPTT memory, hyperparam sensitive.
R2D2, Agent57 use. Simple Atari-এ frame stack যথেষ্ট।

Frame skip + frame stack interaction:

Frame skip 4: agent decides every 4 game frames.
Frame stack 4: those 4 decisions-এর last frame each — covering 16 game frames.
Total temporal window: 16 × ~16ms = 256ms. Enough for ball trajectory.

Other partial-obs hacks in Atari:

Action repeat (= frame skip): consistency in temporal dimension.
Sticky actions: 25% chance of repeating prev action — prevent overfitting to ALE determinism.
No-op start: 1-30 random no-ops at episode start — prevent memorization.

Games where frame stack still insufficient:

Montezuma's Revenge — info hidden behind doors. Long memory required. DQN famously fails.
Pitfall — same issue.
These needed Agent57's intrinsic motivation + memory.

মূল উপলব্ধি: Atari "fully observable" বলা সরল — আসলে partial observability everywhere. Frame stacking — pragmatic solution. RL system design-এ partial-obs সবসময় check.

প্র ০২ Reward clipping ([-1, +1]) Breakout-এ score খুব আলাদা ভাবে scale করে। Bug না কি feature?

চমৎকার trade-off question. DeepMind নিজে paper-এ acknowledge করেছে।

Setup: Breakout brick-এ +1 to +7 (depending on row). 49 games, scales vary wildly — Pong $\pm$1, Space Invaders 5-200, Q*bert 25-1000.

Why clip ([-1, +1]):

Cross-game uniformity: single hyperparameter set works across 49 games.
Q-value stability: Q ranges similar — gradient scale consistent.
Optimization: Adam-এর adaptive learning rate scale-sensitive. Wild reward → wild Q → unstable.
Loss clipping correlation: Huber loss + reward clipping — bounded TD error.

Information loss:

Breakout-এ top-row brick (+7) = bottom-row (+1) — same +1. Strategy difference erased.
"Tunnel trick" — top rows worth more — clipped DQN doesn't directly know.
Despite this, DQN still discovers tunnel — emergent due to sparse-reward dynamics.

Implications:

Optimal clipped policy ≠ optimal unclipped policy.
Breakout's "true score" reaches 800+ — clipped DQN achieves ~400 average.
For some games (Q*bert), clipping limits ceiling significantly.

Modern alternatives:

Reward normalization: running mean/std normalize. Per-game adaptive.
Distributional RL (C51, QR-DQN): learn full reward distribution, not expected value. Naturally handles scale.
Symlog (Dreamer V3): $\text{symlog}(r) = \text{sign}(r) \log(1 + |r|)$ — invertible, scale-invariant.
PopArt (Hessel ২০১৯): adaptive output normalization.

Game-specific tuning:

Single-game research — often skip clipping, use raw rewards.
Multi-game (universal agent) — clipping or normalization essential.

Empirical study (van Hasselt ২০১৬):

Reward clipping responsible for ~30% of DQN's gap to human.
PopArt + DQN — Q*bert 5x improvement.
Distributional RL — even more gain.

মূল কথা: Reward clipping pragmatic engineering choice, not principled. ২০১৩ DQN's "feature" enabling 49-game generality. ২০২৪ — better alternatives exist, used in Rainbow, Dreamer, IMPALA. Atari DQN paper — historical landmark; modern code uses normalization.

প্র ০৩ "Tunnel trick" Breakout-এ — agent ball-কে wall-এর উপর গিয়ে long bounce করে। শেখার পেছনে কী mechanism?

চমৎকার emergent-behavior question.

What's the trick: Breakout-এর ৬ rows of bricks-এর একপাশে repeated hits করে hole ("tunnel") খুলুন। Then ball wall-এর উপরে চলে যায় — top থেকে bricks consecutive break, paddle-এর কাজ নেই for long time. Score-explode.

How DQN discovers it:

Initial random: agent random play, occasional brick hit, +1 reward.
Q-network learns: "hit ball" → reward gradient signal.
Side preference emerges: by chance, agent occasionally hits same column repeatedly. Long reward streak observed.
Q-table update: states leading to "side cluster" — high Q. Behavior reinforces.
Tunneling becomes intentional: 50M+ frames — agent actively aims one column.

Why DQN can find this:

Off-policy + replay — can recall rare lucky long-bounce streaks.
Bootstrap — Q-values propagate "tunnel state is high-value".
Discount factor 0.99 — long-term reward credited.
ε-exploration — eventually stumbles on the strategy.

Why early DQN couldn't (small budget):

10M frames — agent reaches "decent paddle play" but not strategy.
50M-150M — tunnel emerges across most seeds.
Variability across seeds — some seeds find it 30M, others 100M+.

Significance:

Mnih ২০১৫ paper highlighted this — "DQN discovers strategies humans use".
Counter-evidence to "RL = brute memorization" critique.
But: "discover" overstates — chance + reward gradient + persistence.

Modern algorithms — faster discovery:

Rainbow DQN — tunnel by 30M frames.
MuZero — tunnel by 5-10M frames. Tree search efficient exploration.
EfficientZero — tunnel by 1-2M frames. SSL augmentation.

Failure modes:

Some seeds — agent stuck "boring" paddle play, never tunnels.
If reward not clipped (top bricks worth 7 vs bottom 1) — incentive stronger.
If frame skip too aggressive — agent miss the precision required.

Implications for general RL:

Long-horizon credit assignment hard — but possible with patience + correct algorithm.
Exploration exploitation balance — too greedy, miss tunnel; too random, never converge.
Architectural inductive bias minimal — pure CNN + Q-learning sufficient.

মূল কথা: Tunnel trick — "DQN actually learns" demonstration. Not human-level intelligence — but emergent strategic depth from simple rules + lots of practice. Magic trick: nothing in code says "find tunnel"; it emerges.

প্র ০৪ Bangladesh startup-এ Atari-DQN-style RL apply করে কী useful product বানানো যায়?

চমৎকার entrepreneurial question.

Reality check first:

Atari DQN নিজে — academic benchmark, no direct product.
কিন্তু underlying techniques (CNN + Q-learning + replay) — many applications.

Product ideas:

(১) Educational gaming AI:

Bangladeshi school-children's chess/checkers/ludo training opponent.
Adaptive difficulty — student-এর level match.
RL-trained agents at multiple skill levels.
Mobile app, low-bandwidth.

(২) Cricket strategy AI (Bangladesh's love):

State: current match situation (overs, runs, wickets, pitch).
Action: bowl change, fielding setup, batting order.
Reward: probability of winning.
Trained on historical match data + simulator.
Coaches, fantasy leagues, broadcasters use.

(৩) Garment factory cutting optimization:

Atari pixel = fabric pattern image.
Action: cutting position, angle.
Reward: -fabric_waste − time_cost.
Computer vision + RL — automated cutting machine.
RMG industry $40B annual — 1% efficiency = $400M.

(৪) Smart traffic light (small-scale):

Pixel-based traffic camera observation.
Action: phase change.
Reward: throughput - waiting time.
Single intersection prototype → multi-intersection scale.

(৫) Aquaculture (fish/shrimp farming):

Camera observation — fish health, behavior, feed distribution.
Action: feed amount, pump on/off, oxygenation.
Reward: growth rate − cost − mortality.
Bangladesh-er chitoler, golda, vetki industry.

(৬) Educational drone/robot:

Robotics kit + DQN training — STEM education.
Sell to private schools, BUET-NSU labs.

(৭) Game-based therapy:

Stroke recovery, cognitive assessment.
RL-driven adaptive game difficulty.
Healthcare partnership.

(৮) Sports analytics-as-a-service:

BPL teams subscribe — opponent analysis, formation suggestion.
Cricket data + soccer (BPL football).

Practical considerations:

Compute: AWS/GCP credits — startup-এ apply. Lambda Labs cheaper.
Data: domain-specific data hard to gather — partnerships with industry essential.
Productization: RL research demo → robust software 10× engineering.
Customer education: "AI" sells, but RL-specific value unclear. Marketing key.

Realistic startup path:

Phase 1: Solve a clear, narrow industry problem (RMG cutting, aquaculture).
Phase 2: Validate ROI (~10-30% efficiency gain measurable).
Phase 3: Scale to similar industries.
Phase 4: Platform play.

Examples nearby:

India: Niramai (cancer detection RL/CV), Brain.ai.
Bangladesh: ACI Agro AI initiatives, BJIT, Brain Station 23.
Atari-style breakthroughs rare; applied RL niches grow.

মূল কথা: Atari DQN — foundational training ground. Real product-গুলো — domain-specific applications-এ DQN/PPO/SAC apply. Bangladesh-এ untapped opportunities অনেক — domain expertise + ML talent-এর intersection.

অনুশীলন

Pong with smaller compute: Pong-এ DQN train করুন (Breakout-এর চেয়ে easier). ১M frames-এ converge হয় কি?

Pong DQN ১-৫M frames-এ -21 (random) থেকে +21 (perfect) পৌঁছে যায়। Single GPU-এ ২-৪ ঘণ্টা। কারণ — single ball, simple reward (+1 score, -1 lose), no strategy hidden.
Hyperparameter ablation: Frame skip 1 (no skip), 4 (default), 16 try. কোনটা best?

Frame skip 1: 4× compute, no benefit. Frame skip 4: standard, best balance. Frame skip 16: agent miss precise actions (paddle hit), performance drops. ৪ এই sweet spot Atari games-এর জন্য — temporal granularity vs compute trade-off.
Modern improvement: Double DQN implement — single line change. কী?
```
# Original DQN:
q_next = target(SN).max(1).values
# Double DQN:
a_argmax = online(SN).argmax(1)
q_next = target(SN).gather(1, a_argmax.unsqueeze(1)).squeeze(1)
```
Action selection online network থেকে, value evaluation target network থেকে — decouple — overestimation bias কমায়।