পাঠ ১৮ · ৩২-এর মধ্যে · মডিউল ৩
Home / AI Courses / রিইনফোর্সমেন্ট লার্নিং / Actor-Critic

Actor-Critic — দুটি মাথার RL

Actor-Critic — combining policy gradient with value learning
৭ মিনিট পড়া উচ্চ · Advanced PyTorch

এই পাঠে যা শিখবেন

  • Actor ও critic-এর roles
  • Advantage estimation — variance reduction
  • One-step Actor-Critic — full algorithm
  • PyTorch CartPole implementation

১ · REINFORCE-এর সমস্যা থেকে Actor-Critic-এ

REINFORCE — Monte Carlo, episode শেষ পর্যন্ত wait। Actor-Critic — TD-style online update।

Insight: $G_t$-এর জায়গায় $r_t + \gamma V(s_{t+1})$ — single transition থেকে estimate।

২ · Two networks

  • Actor $\pi_\theta(a|s)$: action selection।
  • Critic $V_\phi(s)$: baseline / advantage estimate।

প্রায়ই — shared backbone (CNN/MLP), separate heads।

Advantage estimate

$A(s_t, a_t) = G_t - V(s_t)$ — Monte Carlo।
$A(s_t, a_t) \approx r_t + \gamma V(s_{t+1}) - V(s_t)$ — TD (one-step)।
$A^{n}_t = r_t + \gamma r_{t+1} + \ldots + \gamma^{n-1} r_{t+n-1} + \gamma^n V(s_{t+n}) - V(s_t)$ — n-step।

৩ · Actor-Critic update rules

Actor (policy gradient):

$$\theta \leftarrow \theta + \alpha_\theta \cdot \nabla_\theta \log \pi_\theta(a_t|s_t) \cdot A_t$$

Critic (TD):

$$\phi \leftarrow \phi + \alpha_\phi \cdot (r_t + \gamma V_\phi(s_{t+1}) - V_\phi(s_t)) \cdot \nabla_\phi V_\phi(s_t)$$

বা, equivalently, MSE loss: $\mathcal{L}_\text{critic} = (r_t + \gamma V_\phi(s_{t+1}) - V_\phi(s_t))^2$।

৪ · Why actor-critic better

  • Variance reduction: baseline $V$ subtract — REINFORCE-এর high variance fix।
  • Online: per-step update — episode-শেষ wait নেই।
  • Continuing tasks: works (REINFORCE doesn't)।
  • Continuous control: actor handle।
  • Bias accept: TD bias variance reduction-এর বিনিময়ে — usually win।
Actor-Critic Architecture State s_t observation Backbone shared NN Actor π_θ action distribution Critic V_φ scalar value Action a_t sample π Env r, s' TD error δ δ = r + γV(s') − V(s) r, s' ∇φ V loss δ × ∇θ log π Critic δ থেকে শেখে; Actor δ × ∇log π দিয়ে update। প্রতি step-এ both। এই dual learning structure RL-এর backbone — A2C, PPO, SAC সব এর variant।
Actor-Critic-এর full architecture — actor ও critic shared backbone থেকে আলাদা head, environment feedback দিয়ে both update।

৫ · CartPole Actor-Critic — PyTorch

Python · Actor-Critic 
import gym
import torch
import torch.nn as nn
import torch.optim as optim
from torch.distributions import Categorical
import numpy as np

env = gym.make("CartPole-v1")
state_dim, n_actions = env.observation_space.shape[0], env.action_space.n
gamma = 0.99

class ActorCritic(nn.Module):
    def __init__(self):
        super().__init__()
        self.shared = nn.Sequential(
            nn.Linear(state_dim, 128), nn.ReLU(),
        )
        self.actor = nn.Linear(128, n_actions)
        self.critic = nn.Linear(128, 1)
    def forward(self, s):
        feat = self.shared(s)
        return Categorical(logits=self.actor(feat)), self.critic(feat).squeeze(-1)

model = ActorCritic()
opt = optim.Adam(model.parameters(), lr=3e-4)

for ep in range(500):
    state, _ = env.reset()
    G_total = 0
    while True:
        s_t = torch.FloatTensor(state)
        dist, V_s = model(s_t)
        a = dist.sample()
        log_prob = dist.log_prob(a)
        next_state, r, term, trunc, _ = env.step(a.item())
        done = term or trunc
        s_next = torch.FloatTensor(next_state)
        with torch.no_grad():
            _, V_next = model(s_next)
            V_target = r + gamma * V_next * (0 if done else 1)

        # TD error / advantage
        td_err = V_target - V_s

        # Losses
        actor_loss = -(log_prob * td_err.detach())
        critic_loss = td_err.pow(2)
        loss = actor_loss + critic_loss

        opt.zero_grad()
        loss.backward()
        opt.step()

        G_total += r
        state = next_state
        if done: break
    if ep % 25 == 0:
        print(f"Ep {ep:3d} | return = {G_total:.1f}")

    


    

      ১৫০-৩০০ episode-এ converge — REINFORCE-এর চেয়ে দ্রুত। variance reduction-এর সরাসরি effect।
    


    
৬ · One-step bias problem


    

      One-step TD bias আনে — $V(s')$ approximate। Solutions:
    

    
    
      
  • n-step return — partial Monte Carlo।
    • 
      
  • GAE (Schulman 2016) — বিভিন্ন n-এর exponential average।
    • 
      
  • $\lambda$-return.
    • 
    


    
৭ · Generalized Advantage Estimation (GAE)


    

      $$A^{GAE(\lambda)}_t = \sum_{k=0}^\infty (\gamma \lambda)^k \delta_{t+k}$$
    


    

      $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$ — TD error।
    


    

      $\lambda = 0$ — pure TD। $\lambda = 1$ — Monte Carlo। typical $\lambda = 0.95$।
    


    
৮ · Variants ও extensions


    
    
      
  • A2C (Advantage Actor-Critic): synchronous parallel rollout।
    • 
      
  • A3C: asynchronous parallel।
    • 
      
  • PPO: A2C + clipped objective।
    • 
      
  • SAC: off-policy, max-entropy।
    • 
      
  • DDPG/TD3: deterministic actor, off-policy।
    • 
      
  • IMPALA: distributed AC, scalable।
    • 
    


    
৯ · কোথায় ব্যবহার


    
    
      
  • Atari (A2C, A3C, IMPALA)।
    • 
      
  • Continuous control (PPO, SAC)।
    • 
      
  • Robotics।
    • 
      
  • RLHF (PPO)।
    • 
      
  • StarCraft II (AlphaStar — IMPALA + auxiliary)।
    • 
    


    

      Modern deep RL = mostly actor-critic family।
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      প্র ০১Actor ও critic-এর shared backbone vs separate networks — trade-off?
      

        

          Shared backbone:
        

        
    
          
  • Pros: parameter efficient, feature reuse, less overfit।
    • 
          
  • Cons: gradient interference (actor pulls one way, critic অন্য)।
    • 
        

        

          Separate:
        

        
    
          
  • Pros: stable, isolated learning।
    • 
          
  • Cons: 2x parameters, slower।
    • 
        

        

          Practical:
        

        
    
          
  • Atari (image input) — shared CNN backbone, separate heads — sample efficient।
    • 
          
  • MuJoCo (low-D continuous) — separate networks-এ বেশি stable।
    • 
          
  • RLHF — completely separate (actor = LLM, critic = small MLP)।
    • 
        

        

          Loss balancing: shared backbone-এ — coefficient $c_1, c_2$ critical। $\mathcal{L} = \mathcal{L}_{actor} + c_1 \mathcal{L}_{critic} + c_2 \mathcal{L}_{entropy}$।
        

      

    


    

      প্র ০২td_err.detach() কেন actor loss-এ critical?
      

        

          Code দেখুন: actor_loss = -(log_prob * td_err.detach())।
        

        

          কেন detach:
        

        
    
          
  • td_err = $r + \gamma V(s') - V(s)$ — depends on critic parameters।
    • 
          
  • Detach ছাড়া — gradient td_err হয়ে critic-কেও modify করত actor loss-এর কারণে।
    • 
          
  • Conceptually wrong — actor critic-কে inform করতে চায়, modify করতে না।
    • 
        

        

          Two separate updates:
        

        
    
          
  • Critic: TD error squared → $\nabla_\phi$।
    • 
          
  • Actor: log_prob × (TD error treated as constant) → $\nabla_\theta$।
    • 
        

        

          Common bug: detach না করলে — training unstable, sometimes diverge।
        

        

          Numerical alternative: $\mathcal{L} = \mathcal{L}_a(\theta) + \mathcal{L}_c(\phi)$ separately compute, separate optimizer। code clean কিন্তু ২x param dict।
        

      

    


    

      প্র ০৩Why entropy regularization in actor-critic?
      

        

          Modern actor-critic — entropy bonus যোগ:
        

        

          $\mathcal{L} = \mathcal{L}_{actor} + c \mathcal{L}_{critic} - \beta \mathcal{H}(\pi(\cdot|s))$
        

        

          $\mathcal{H} = -\sum_a \pi(a|s) \log \pi(a|s)$ — entropy।
        

        

          কেন:
        

        
    
          
  • Encourages exploration — uniform-এর কাছাকাছি policy।
    • 
          
  • Premature convergence prevent।
    • 
          
  • Local optima escape।
    • 
        

        

          Empirical:
        

        
    
          
  • $\beta = 0.01$ Atari standard।
    • 
          
  • SAC — entropy IS the objective (max-ent RL)।
    • 
          
  • RLHF — entropy বদলে KL penalty (base model থেকে drift control)।
    • 
        

        

          Adaptive entropy: SAC auto-tune $\beta$ — target entropy specify।
        

      

    


    

      প্র ০৪Actor-critic কেন on-policy থাকে — off-policy version সম্ভব?
      

        

          Vanilla A2C — on-policy। সাম্প্রতিক data current policy-র দরকার।
        

        

          Off-policy AC সম্ভব:
        

        
    
          
  • SAC: Q-function critic + replay buffer। off-policy।
    • 
          
  • DDPG/TD3: deterministic actor + Q critic + replay।
    • 
          
  • ACER: on-policy A3C + replay (importance sampling correct)।
    • 
          
  • IMPALA: V-trace correction।
    • 
        

        

          Off-policy AC-এর challenges:
        

        
    
          
  • Importance sampling variance।
    • 
          
  • Distribution shift।
    • 
          
  • Stability tricks दरকার।
    • 
        

        

          Why on-policy popular:
        

        
    
          
  • Simpler — no IS correction।
    • 
          
  • Stable theoretically।
    • 
          
  • PPO — on-policy, dominant in research।
    • 
        

        

          Trend: sample efficiency-প্রিয় domain-এ off-policy (SAC), simulator-rich domain-এ on-policy (PPO)।
        

      

    

  


  

    
 অনুশীলন

    
    
      
  1. 
        Compute advantage: $V(s)=4, r=2, V(s')=5, \gamma=0.9$, not done। one-step advantage?
        
    
          
          
    
            
    $A = r + \gamma V(s') - V(s) = 2 + 4.5 - 4 = 2.5$।
    
          
    
        
    
      
    2. 

      
  2. 
        GAE: $\delta_0=2, \delta_1=1, \delta_2=-1$, $\gamma=0.9, \lambda=0.95$। $A^{GAE}_0$?
        
    
          
          
    
            
    $A_0^{GAE} = \delta_0 + (\gamma\lambda)\delta_1 + (\gamma\lambda)^2\delta_2 = 2 + 0.855 \cdot 1 + 0.731 \cdot (-1) = 2 + 0.855 - 0.731 = 2.124$।
    
          
    
        
    
      
    3. 

      
  3. 
        Code modify: উপরের actor-critic-এ entropy bonus যোগ করুন।
        
    
          
          
    
            
    entropy = dist.entropy().mean()
loss = actor_loss + critic_loss - 0.01 * entropy
    
            
    সাধারণত exploration improve, learning স্থিতিশীল।
    
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন

    
  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ১৭ · REINFORCE