পাঠ ২২ · ৩২-এর মধ্যে · মডিউল ৩
Home / AI Courses / রিইনফোর্সমেন্ট লার্নিং / SAC

SAC — Soft Actor-Critic

SAC — maximum entropy continuous control
৭ মিনিট পড়া উচ্চ · Advanced PyTorch

এই পাঠে যা শিখবেন

  • Maximum entropy RL framework
  • SAC architecture — actor + 2 Q-networks + targets
  • Reparameterization trick — continuous action backprop
  • Auto-tuned α

১ · Maximum entropy RL

Standard RL: $J = \mathbb{E}[\sum r_t]$।
Max-entropy RL: $J = \mathbb{E}[\sum r_t + \alpha \mathcal{H}(\pi(\cdot|s_t))]$।

$\alpha$ — temperature। entropy bonus encourages diversity।

কেন entropy bonus

১) Exploration: stochastic policy, all action positive probability।
২) Robustness: multimodal optimal — sub-optimal converge avoid।
৩) Transfer: diverse policy — task variation-এ adapt।
৪) Stability: deterministic-এর চেয়ে training smoother।

২ · Soft Q-function

$$Q_{soft}^\pi(s, a) = \mathbb{E}\left[ \sum_t \gamma^t (r_t + \alpha \mathcal{H}(\pi(\cdot|s_t))) \right]$$

Bellman backup:

$$Q(s, a) = r + \gamma \mathbb{E}_{s'} [V(s')]$$

$$V(s) = \mathbb{E}_{a \sim \pi}[Q(s, a) - \alpha \log \pi(a|s)]$$

Notice — $V$-এ entropy term।

৩ · Optimal stochastic policy

Soft RL-এ optimal policy energy-based:

$$\pi^*(a|s) \propto \exp\left( \frac{1}{\alpha} Q^*(s, a) \right)$$

$\alpha \to 0$: deterministic optimal (Boltzmann)।
$\alpha \to \infty$: uniform random।

৪ · SAC architecture

  • Actor $\pi_\theta(a|s)$ — Gaussian (continuous)।
  • Q-networks $Q_{\phi_1}, Q_{\phi_2}$ — twin (overestimation reduce)।
  • Target Q-networks $Q_{\bar{\phi}_1}, Q_{\bar{\phi}_2}$ — Polyak average।
  • Replay buffer — off-policy।
  • α (entropy coef) — learnable parameter।

৫ · Critic update

$$\mathcal{L}_Q(\phi_i) = \mathbb{E}_{(s, a, r, s') \sim D} \left[ (Q_{\phi_i}(s, a) - y)^2 \right]$$

$$y = r + \gamma \left( \min_{j=1,2} Q_{\bar{\phi}_j}(s', \tilde{a}') - \alpha \log \pi_\theta(\tilde{a}' | s') \right), \quad \tilde{a}' \sim \pi_\theta(\cdot | s')$$

Twin trick: min of two targets — overestimation cap।

৬ · Actor update — reparameterization

Gradient through stochastic action-এর জন্য reparameterization:

$\tilde{a} = f_\theta(\xi; s) = \mu_\theta(s) + \sigma_\theta(s) \cdot \xi$, $\xi \sim \mathcal{N}(0, 1)$।

Actor loss:

$$\mathcal{L}_\pi(\theta) = \mathbb{E}_{s, \xi} \left[ \alpha \log \pi_\theta(\tilde{a}|s) - \min_j Q_{\phi_j}(s, \tilde{a}) \right]$$

৭ · Auto-tune α

Original SAC — manual α। Auto-α (Haarnoja 2018b):

$$\mathcal{L}_\alpha = -\alpha \cdot \mathbb{E}\left[ \log \pi(a|s) + \bar{H} \right]$$

Target entropy $\bar{H}$ — task-specific (e.g., $-|A|$ for $|A|$-D continuous)।

Effect: $\alpha$ adapt — exploration বেশি হলে decrease, কম হলে increase।

SAC — Soft Actor-Critic Actor π_θ Gaussian(μ, σ) reparameterization Q_φ₁ critic 1 Q_φ₂ critic 2 (twin) Q̄_φ̄₁ target (Polyak) Q̄_φ̄₂ target (Polyak) min target y α (auto-tuned) entropy coef Replay Buffer off-policy data Twin Q + entropy regularization + reparameterization → SAC। MuJoCo continuous control SOTA।
SAC architecture — actor + twin critics + targets + α + replay। DDPG-এর সব lesson learned-এর synthesis।

৮ · PyTorch SAC — minimal

Python · SAC core 
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.distributions import Normal

class GaussianActor(nn.Module):
    def __init__(self, state_dim, action_dim):
        super().__init__()
        self.shared = nn.Sequential(
            nn.Linear(state_dim, 256), nn.ReLU(),
            nn.Linear(256, 256), nn.ReLU(),
        )
        self.mean = nn.Linear(256, action_dim)
        self.log_std = nn.Linear(256, action_dim)

    def forward(self, s):
        f = self.shared(s)
        mu = self.mean(f)
        log_std = self.log_std(f).clamp(-20, 2)
        std = log_std.exp()
        dist = Normal(mu, std)
        # Reparameterization
        z = dist.rsample()
        # Squash to [-1, 1] via tanh
        a = torch.tanh(z)
        # Log prob with tanh correction
        log_prob = dist.log_prob(z) - torch.log(1 - a.pow(2) + 1e-6)
        log_prob = log_prob.sum(-1, keepdim=True)
        return a, log_prob

class QNetwork(nn.Module):
    def __init__(self, state_dim, action_dim):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(state_dim + action_dim, 256), nn.ReLU(),
            nn.Linear(256, 256), nn.ReLU(),
            nn.Linear(256, 1),
        )
    def forward(self, s, a):
        return self.net(torch.cat([s, a], dim=-1)).squeeze(-1)

def sac_critic_loss(Q1, Q2, Q1_targ, Q2_targ, actor, batch, alpha, gamma=0.99):
    s, a, r, s_next, d = batch
    with torch.no_grad():
        a_next, log_p_next = actor(s_next)
        q_next = torch.min(Q1_targ(s_next, a_next), Q2_targ(s_next, a_next))
        target = r + gamma * (1 - d) * (q_next - alpha * log_p_next.squeeze(-1))
    L1 = F.mse_loss(Q1(s, a), target)
    L2 = F.mse_loss(Q2(s, a), target)
    return L1 + L2

def sac_actor_loss(Q1, Q2, actor, s, alpha):
    a, log_p = actor(s)
    q = torch.min(Q1(s, a), Q2(s, a))
    return (alpha * log_p.squeeze(-1) - q).mean()

print("SAC core ready। Replay buffer + Polyak update + α tuning add করলে full SAC।")

    


    

      Full SAC implementation আরও সাজাতে হবে (training loop, target update, replay)। Stable-Baselines3-এর SAC ~৩০০ lines, production-ready।
    


    
৯ · SAC vs PPO — কোনটি কখন


    

      

        Criterion
        SAC
        PPO
      

      

        Type
        Off-policy
        On-policy
      

      

        Sample efficiency
        High
        Lower
      

      

        Action space
        Continuous (preferred)
        Both
      

      

        Robustness
        High (entropy)
        Hyperparameter sensitive
      

      

        Implementation
        Complex (5+ networks)
        Simple
      

    


    

      Best fit:
    

    
    
      
  • SAC: real robot, expensive sample, MuJoCo।
    • 
      
  • PPO: cheap simulator (Atari, parallel envs), RLHF।
    • 
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      প্র ০১"Reparameterization trick" কেন SAC-এ critical?
      

        

          Stochastic action-এর gradient — direct backprop impossible: $a \sim \pi_\theta(\cdot|s)$ এর "sample" operation differentiable না।
        

        

          Reparameterization: $a = f_\theta(\xi; s)$, $\xi \sim p(\xi)$ (fixed)।
        

        

          Gaussian-এ: $a = \mu_\theta(s) + \sigma_\theta(s) \cdot \xi$, $\xi \sim \mathcal{N}(0, 1)$।
        

        

          Effect: $\theta$-এ gradient flow — $\mu, \sigma$-এ derivative। $\xi$ external noise।
        

        

          SAC-এ ব্যবহার: $\mathcal{L}_\pi = \mathbb{E}_\xi[\alpha \log \pi(f_\theta(\xi)) - Q(s, f_\theta(\xi))]$। $\theta$ via $f_\theta$ propagate।
        

        

          Alternative — log-derivative (REINFORCE): high variance। SAC-এ unsuitable।
        

        

          Tanh squashing: action bounded $[-1, 1]$। log_prob-এ correction term জরুরি (Jacobian)।
        

      

    


    

      প্র ০২Twin Q networks — DDPG-এ overestimation কেন এত vicious?
      

        

          DDPG-এর target: $r + \gamma Q_{\bar{\phi}}(s', \mu_\theta(s'))$।
        

        

          সমস্যা:
        

        
    
          
  • $\mu_\theta(s')$ — actor "best" action choose।
    • 
          
  • Q noisy — actor over-estimated action select।
    • 
          
  • Q overestimate amplified — actor সেই inflated action follow।
    • 
          
  • Feedback loop। divergence।
    • 
        

        

          TD3 (২০১৮) fix:
        

        
    
          
  • Twin Q networks — min target।
    • 
          
  • Delayed actor update (every 2 critic steps)।
    • 
          
  • Target action smoothing (noise add)।
    • 
        

        

          SAC inherit: twin Q, soft update।
        

        

          Empirical: twin Q — single Q-এর তুলনায় MuJoCo-এ ৫০-৭০% better return।
        

        

          মূল উপলব্ধি: "actor follow Q" feedback — Q overestimation সরাসরি actor-এ propagate। min trick conservative target।
        

      

    


    

      প্র ০৩Auto-tuned α-এর target entropy choice কীভাবে?
      

        

          Target entropy $\bar{H}$ — desired minimum entropy। Action-space-নির্ভর:
        

        
    
          
  • $|A|$-D continuous: $\bar{H} = -|A|$ (default heuristic)।
    • 
          
  • Discrete: $\bar{H} = 0.98 \cdot \log |A|$ (close to uniform)।
    • 
        

        

          α update intuition:
        

        
    
          
  • Current entropy > target — α decrease (less exploration)।
    • 
          
  • Current entropy < target — α increase (more exploration)।
    • 
        

        

          Math:
        

        

          $\mathcal{L}_\alpha = -\alpha \cdot (\log \pi + \bar{H})$।
        

        

          $\nabla_\alpha = -(log \pi + \bar{H})$।
        

        
    
          
  • $\log \pi + \bar{H} > 0$ — entropy "above target", α decrease।
    • 
          
  • $\log \pi + \bar{H} < 0$ — entropy "below target", α increase।
    • 
        

        

          Practical: $\log \alpha$ optimize (positivity ensure)। typical 1e-3 to 0.5 range during training।
        

      

    


    

      প্র ০৪SAC sample efficiency PPO-এর তুলনায় কেন এত better?
      

        

          MuJoCo HalfCheetah-এ:
        

        
    
          
  • SAC: 1M steps-এ ~10K return।
    • 
          
  • PPO: 5M+ steps to similar।
    • 
        

        

          কেন SAC better sample-wise:
        

        
    
          
  • Off-policy: replay reuse — প্রতি transition বহুবার gradient।
    • 
          
  • Twin Q + target net: stable estimate। PPO-এর critic less stable।
    • 
          
  • Entropy regularization: consistent exploration। PPO entropy bonus ad-hoc।
    • 
          
  • Continuous control optimal: Gaussian policy + Q-function — dense gradient signal।
    • 
        

        

          কেন PPO better wall-clock:
        

        
    
          
  • Parallel rollout — many env একসাথে।
    • 
          
  • Simpler — fewer networks।
    • 
          
  • GPU-friendly batch।
    • 
        

        

          Real-world preference:
        

        
    
          
  • Robot (sample expensive): SAC।
    • 
          
  • Simulator (sample cheap): PPO।
    • 
        

      

    

  


  

    
 অনুশীলন

    
    
      
  1. 
        Soft V derivation: $Q(s, a) = 5$, $\log \pi(a|s) = -1$, $\alpha = 0.2$। $V_{soft}(s)$ contribution from this $a$?
        
    
          
          
    
            
    $\pi$-weighted: $V = E[Q - \alpha \log \pi] = 5 - 0.2 \cdot (-1) = 5.2$ (single action contribution)।
    
          
    
        
    
      
    2. 

      
  2. 
        Target entropy: 6-D continuous control। recommended target entropy?
        
    
          
          
    
            
    $\bar{H} = -|A| = -6$।
    
          
    
        
    
      
    3. 

      
  3. 
        Code: উপরের SAC actor-এ — log_prob-এর tanh correction কেন দরকার?
        
    
          
          
    
            
    $a = \tanh(z)$ — change of variable। density: $p_a(a) = p_z(z) / |\partial a/\partial z|$। $\partial a/\partial z = 1 - a^2$।
    
            
    $\log p_a = \log p_z - \log(1 - a^2)$। numerical stability-এর জন্য $+1e-6$।
    
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন

    
  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ২১ · PPO