পাঠ ১৯ · ৩২-এর মধ্যে · মডিউল ৩

Home / AI Courses / রিইনফোর্সমেন্ট লার্নিং / A2C & A3C

A2C ও A3C — Parallel Actor-Critic

A2C & A3C — synchronous and asynchronous parallel actor-critic

৭ মিনিট পড়া উচ্চ · Advanced PyTorch

এই পাঠে যা শিখবেন

A3C — asynchronous parallelism কেন কাজ করে
A2C — সরল synchronous version
GAE — Generalized Advantage Estimation
Vector environment-এ multi-worker rollout

১ · A3C (Mnih et al., 2016)

DQN-এর alternative হিসেবে DeepMind A3C বানালো — replay buffer ছাড়া, parallel workers দিয়ে decorrelation।

Key insight: N workers each its own env। concurrent rollout — different state visited। অনেকটা replay buffer-এর diverse data।

২ · Architecture

Central global network ($\theta_{global}$)।
$N$ workers, each:
- Own env copy।
- Local copy of network।
- Run rollout for $T_{max}$ steps।
- Compute gradient।
- Apply gradient to global (asynchronous, lock-free)।
- Sync local network with global।

৩ · Asynchronous lock-free SGD

Multiple workers same parameters update — race condition। কিন্তু empirically — Hogwild-style (Niu et al., 2011) lock-free SGD-এ noisy gradient acts like regularization।

A3C-এর ৩-letter — Asynchronous Advantage Actor-Critic।

৪ · A2C — synchronous variant

OpenAI (Wu et al., 2017) — A3C-এর synchronous version "A2C" । সব workers wait, batched update।

সুবিধা:

GPU friendly — batch norm, large matmul।
More stable — no race conditions।
Easier debug।
Empirically — A3C-এর সমান বা better performance।

A3C-এর "asynchronous" advantage actually overrated (in retrospect) — A2C-এর simpler ভাল।

A3C-এর architecture — central global network, multiple parallel workers, asynchronous gradient push।

৫ · GAE — Generalized Advantage Estimation

Schulman et al. (2016)। One-step TD biased, MC noisy। GAE — exponentially weighted intermediate:

$$\hat{A}_t^{GAE(\lambda)} = \sum_{l=0}^\infty (\gamma \lambda)^l \delta_{t+l}$$

$\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$।

$\lambda = 0$ — TD(0)। $\lambda = 1$ — Monte Carlo। typical $\lambda = 0.95$।

৬ · A2C — full PyTorch (vector env)

Python · A2C with vector envs

import gym
import torch
import torch.nn as nn
import torch.optim as optim
from torch.distributions import Categorical
import numpy as np

# Vector env — N parallel envs in lockstep
def make_envs(n=8):
    return gym.vector.SyncVectorEnv([lambda: gym.make("CartPole-v1") for _ in range(n)])

envs = make_envs(8)
state_dim, n_actions = 4, 2

class ActorCritic(nn.Module):
    def __init__(self):
        super().__init__()
        self.shared = nn.Sequential(nn.Linear(state_dim, 128), nn.ReLU())
        self.actor = nn.Linear(128, n_actions)
        self.critic = nn.Linear(128, 1)
    def forward(self, s):
        f = self.shared(s)
        return Categorical(logits=self.actor(f)), self.critic(f).squeeze(-1)

model = ActorCritic()
opt = optim.Adam(model.parameters(), lr=3e-4)
gamma, lam = 0.99, 0.95
T = 8  # rollout length

states, _ = envs.reset()
for update in range(500):
    # Collect rollout
    log_probs, values, rewards, dones = [], [], [], []
    for t in range(T):
        s_t = torch.FloatTensor(states)
        dist, V = model(s_t)
        a = dist.sample()
        log_probs.append(dist.log_prob(a))
        values.append(V)
        next_states, r, term, trunc, _ = envs.step(a.numpy())
        rewards.append(torch.FloatTensor(r))
        dones.append(torch.FloatTensor(term | trunc))
        states = next_states

    # Last bootstrap value
    with torch.no_grad():
        _, V_last = model(torch.FloatTensor(states))

    # GAE advantage backward
    advantages = []
    A = torch.zeros(8)
    V_next = V_last
    for t in reversed(range(T)):
        delta = rewards[t] + gamma * V_next * (1 - dones[t]) - values[t]
        A = delta + gamma * lam * (1 - dones[t]) * A
        advantages.insert(0, A)
        V_next = values[t]

    advantages = torch.stack(advantages)
    log_probs_t = torch.stack(log_probs)
    values_t = torch.stack(values)
    returns = advantages + values_t.detach()

    # Losses
    actor_loss = -(log_probs_t * advantages.detach()).mean()
    critic_loss = (returns - values_t).pow(2).mean()
    loss = actor_loss + 0.5 * critic_loss

    opt.zero_grad()
    loss.backward()
    nn.utils.clip_grad_norm_(model.parameters(), 0.5)
    opt.step()

    if update % 25 == 0:
        avg_r = torch.stack(rewards).sum().item() / 8
        print(f"Update {update:3d} | avg rollout reward = {avg_r:.1f}")

৮ env parallel + GAE — পরে rollout সংগ্রহের পর update। একক env-এর তুলনায় ৬-৮x faster wall-clock।

৭ · Hyperparameters

$N$ (workers): 8-32। CPU core count।
$T$ (rollout): 5-128। long-horizon-এ বেশি, exploration-এ কম।
$\lambda$ (GAE): 0.95 — most tasks।
Entropy coef: 0.01।
Critic coef: 0.5।
Grad clip: 0.5।

৮ · IMPALA — A3C-এর scaling

Espeholt et al. (2018) — IMPALA, distributed-A3C। ১০০০+ actors centralized learner-এ feed। V-trace correction — slight off-policy correction।

AlphaStar (StarCraft II) — IMPALA-based architecture।

৯ · A2C/A3C-এর সীমা

On-policy: sample inefficient।
Step size sensitive: bad update — policy collapse।
Hyperparameter tuning: entropy coef, lr, T — tune-heavy।

পরের পাঠ — TRPO, PPO — এই step-size সমস্যা address।

ভাবনার প্রশ্ন

প্র ০১A3C-এর "asynchronous" actually critical না?

OpenAI A2C paper (2017) — A3C-এর synchronous version benchmarks-এ better বা equal।

কেন A3C-এর async over-rated:

"Decorrelation" workers থেকে — sync-ও পায়।
Stale gradient (asynchronous-এ) — small bias।
GPU underutilization async-এ — হার্ডওয়্যার waste।

কেন sync better:

Batched matrix multiply on GPU।
Reproducible (deterministic ordering)।
Debug সহজ।
Hogwild noise-এর benefit modest GPU-এ।

আজকের practice: PPO, IMPALA-ও sync-style বা batch-async। pure A3C rare।

ঐতিহাসিক context: 2016-এ GPU expensive, CPU plentiful। তখন A3C ভাল fit। আজ GPU-rich — A2C/PPO win।

প্র ০২GAE-এর λ tuning — practical advice?

$\lambda$ — bias-variance trade-off:

$\lambda = 0$: pure TD — low variance, high bias।
$\lambda = 1$: pure MC — unbiased, high variance।
$\lambda = 0.95$: standard sweet spot।

Task-specific tuning:

Long horizon (Atari, robotics): $\lambda = 0.95-0.99$।
Short horizon: $\lambda = 0.9$।
Sparse reward: $\lambda$ closer to 1 — Bellman propagation।
Noisy reward: $\lambda$ smaller — variance critical।

Implementation note: backward computation efficient — $A_t = \delta_t + \gamma\lambda A_{t+1}$।

Empirical: $\lambda = 0.95, \gamma = 0.99$ — RLLib, Stable-Baselines defaults। most users-এর সরাসরি default ব্যবহার OK।

প্র ০৩Vector env vs separate processes — কোনটি কখন?

SyncVectorEnv: single process, sequential step। সরল কিন্তু slow।

AsyncVectorEnv (multiprocessing): multiple processes, true parallelism।

সবার trade-off:

Env step cheap (CartPole) — sync OK।
Env step heavy (MuJoCo, Atari) — async win।
Image processing-heavy — async crucial।

Hardware:

16+ CPU core: async truly parallel।
Single GPU bottleneck: small batch, sync OK।
Multi-GPU: distributed setup — Ray, Horovod।

Modern: EnvPool (NVIDIA, 2021) — C++ env vector — 100x faster Atari sampling। sample-bottleneck issue largely solved।

প্র ০৪Why A3C/A2C-এর variance কম REINFORCE-এর তুলনায়?

Three sources of variance reduction:

(১) Critic baseline:

Advantage = $G - V(s)$।
$V$-এর around variance — $G$-এর তুলনায় কম।

(২) GAE truncation:

$\lambda < 1$ — long-horizon noise dampened।
Trajectory variance শুধু effective horizon-এর।

(৩) Parallel rollout:

$N$ env-এর diverse data।
Gradient estimate $\bar{g} = \frac{1}{N}\sum g_i$ — variance $1/N$ scaled।

Combined effect: REINFORCE-এর variance vs A3C — সাধারণত ১০-১০০x reduction। এই কারণেই A2C/A3C Atari-এ usable, REINFORCE নয়।

অনুশীলন

GAE compute: $\delta = [1, 2, -1, 0]$, $\gamma=0.99, \lambda=0.95$। $A_0$?

$\gamma\lambda = 0.9405$।

$A_3 = 0$। $A_2 = -1 + 0.9405 \cdot 0 = -1$। $A_1 = 2 + 0.9405 \cdot (-1) = 1.06$। $A_0 = 1 + 0.9405 \cdot 1.06 = 1.997$।
Speed-up estimate: single env-এ episode ৫ sec, ১৬ env vector-এ episode ৬ sec (parallel)। speedup?

Single: 5 sec/episode. Vector: 6 sec for 16 episodes ⇒ 0.375 sec/episode। speedup ≈ 13x।
Code modify: উপরের code-এ $T=32$ rollout করুন। convergence কী হবে?

Longer rollout — better advantage estimate (less bootstrap), more samples per update। usually faster convergence কিন্তু memory বেশি। Atari-এ T=128 standard।