পাঠ ২৩ · ৩২-এর মধ্যে · মডিউল ৩
Home / AI Courses / রিইনফোর্সমেন্ট লার্নিং / DDPG & TD3

DDPG ও TD3 — Deterministic Continuous Control

DDPG & TD3 — deterministic policy gradient for continuous actions
৭ মিনিট পড়া উচ্চ · Advanced PyTorch

এই পাঠে যা শিখবেন

  • Deterministic Policy Gradient (DPG) theorem
  • DDPG architecture ও training loop
  • TD3-এর তিন improvement
  • Continuous control benchmarks

১ · Deterministic policy gradient

Silver et al. (২০১৪) — DPG theorem। Stochastic policy-এর জায়গায় deterministic $\mu_\theta(s)$:

$$\nabla_\theta J(\mu_\theta) = \mathbb{E}_{s \sim \rho^\mu}\left[ \nabla_a Q^\mu(s, a)\big|_{a=\mu(s)} \cdot \nabla_\theta \mu_\theta(s) \right]$$

Chain rule — $a = \mu_\theta(s)$, gradient flow $\theta \to a \to Q$।

২ · DDPG = DPG + DQN tricks

Lillicrap et al. (২০১৬) DeepMind। Continuous control-এর জন্য:

  • Actor $\mu_\theta(s)$ — deterministic, output continuous action।
  • Critic $Q_\phi(s, a)$ — single Q-network।
  • Target nets $\mu_{\bar{\theta}}, Q_{\bar{\phi}}$ — Polyak average।
  • Replay buffer — off-policy data।
  • Action exploration — Ornstein-Uhlenbeck noise (later — Gaussian noise)।

৩ · Critic update

$$y = r + \gamma Q_{\bar{\phi}}(s', \mu_{\bar{\theta}}(s'))$$

$$\mathcal{L}_Q = \mathbb{E}[(Q_\phi(s, a) - y)^2]$$

৪ · Actor update

DPG theorem অনুসারে:

$$\nabla_\theta J = \mathbb{E}\left[ \nabla_a Q_\phi(s, a)\big|_{a=\mu_\theta(s)} \cdot \nabla_\theta \mu_\theta(s) \right]$$

Equivalently — minimize $-Q_\phi(s, \mu_\theta(s))$ — actor try to produce action with high Q।

৫ · DDPG-এর সমস্যা — overestimation

DQN-এর মত — Q overestimate। DDPG-এ আরও worse — actor "best action" follow করে। Q overestimated action select।

Feedback loop:

  • Q noisy → some action over-Q।
  • Actor সেই action follow।
  • Q backup-এ inflated value propagate।
  • Q আরও inflate।

৬ · TD3 — তিন fix

Fujimoto et al. (২০১৮) "Addressing Function Approximation Error":

  1. Twin critic: $Q_{\phi_1}, Q_{\phi_2}$। target = $\min(Q_1, Q_2)$ — overestimation cap।
  2. Delayed actor update: critic update প্রতি step, actor update প্রতি $d$ step (typical $d=2$)। critic stabilize।
  3. Target action smoothing: target action-এ noise — $\tilde{a}' = \mu_{\bar{\theta}}(s') + \epsilon$, $\epsilon \sim \mathcal{N}(0, \sigma)$ clipped। regularization।

Combined target:

$$y = r + \gamma \min_{j=1,2} Q_{\bar{\phi}_j}(s', \mu_{\bar{\theta}}(s') + \text{clip}(\epsilon, -c, c))$$

DDPG vs TD3 — Architecture comparison DDPG (2016) Actor μ_θ Critic Q_φ Target μ̄ Target Q̄ ⚠ Overestimation Q noisy → actor picks high → Q inflate Single Q, simultaneous updates TD3 (2018) Actor (delayed) Q_φ₁ Q_φ₂ (twin) Target μ̄ + noise Target Q̄₁ Target Q̄₂ ✓ Three fixes: 1. min(Q₁, Q₂) target 2. Actor every 2 critic steps 3. Target action smoothing TD3 — DDPG-এর সরাসরি upgrade। MuJoCo benchmarks-এ DDPG ভাঙে, TD3 stable। SAC ও TD3 — দু'টি modern continuous control workhorse।
DDPG ও TD3 architecture — TD3-এর তিন fix। Twin critic + delayed actor + target smoothing।

৭ · TD3 PyTorch core

Python · TD3 update 
import torch
import torch.nn.functional as F

def td3_update(actor, critic1, critic2, actor_targ, critic1_targ, critic2_targ,
               opt_actor, opt_critic, batch, step,
               gamma=0.99, tau=0.005, policy_noise=0.2, noise_clip=0.5,
               policy_delay=2, max_action=1.0):
    s, a, r, s_next, d = batch

    # Target action with smoothing noise
    with torch.no_grad():
        noise = (torch.randn_like(a) * policy_noise).clamp(-noise_clip, noise_clip)
        a_next = (actor_targ(s_next) + noise).clamp(-max_action, max_action)
        q1_targ = critic1_targ(s_next, a_next)
        q2_targ = critic2_targ(s_next, a_next)
        target = r + gamma * (1 - d) * torch.min(q1_targ, q2_targ)

    # Critic update
    q1, q2 = critic1(s, a), critic2(s, a)
    critic_loss = F.mse_loss(q1, target) + F.mse_loss(q2, target)
    opt_critic.zero_grad(); critic_loss.backward(); opt_critic.step()

    # Delayed actor update
    if step % policy_delay == 0:
        # Maximize Q1(s, μ(s))
        actor_loss = -critic1(s, actor(s)).mean()
        opt_actor.zero_grad(); actor_loss.backward(); opt_actor.step()

        # Polyak target updates
        for p, p_t in zip(actor.parameters(), actor_targ.parameters()):
            p_t.data.copy_(tau * p.data + (1 - tau) * p_t.data)
        for p, p_t in zip(critic1.parameters(), critic1_targ.parameters()):
            p_t.data.copy_(tau * p.data + (1 - tau) * p_t.data)
        for p, p_t in zip(critic2.parameters(), critic2_targ.parameters()):
            p_t.data.copy_(tau * p.data + (1 - tau) * p_t.data)

print("TD3 update step ready")

    


    

      Stable-Baselines3-এর TD3 production-ready। MuJoCo Hopper, HalfCheetah, Ant — ১M steps-এ great performance।
    


    
৮ · Exploration in DDPG/TD3


    

      Deterministic policy — exploration external:
    

    
    
      
  • OU noise (DDPG original): temporally correlated noise — physical actuator-এর smooth।
    • 
      
  • Gaussian noise (TD3): simpler — same performance practical।
    • 
      
  • Parameter space noise: network weights-এ noise (NoisyNet)।
    • 
    


    

      Test time-এ — pure deterministic, no noise।
    


    
৯ · DDPG/TD3 vs SAC — final word


    
    
      
  • Performance: SAC ও TD3 most MuJoCo benchmark-এ comparable। SAC slightly better robustness।
    • 
      
  • Hyperparameters: TD3 fewer (no α tuning)।
    • 
      
  • Implementation: TD3 simpler (deterministic)।
    • 
      
  • Theoretical: SAC max-entropy framework cleaner।
    • 
      
  • Practical advice: SAC default choice continuous control। TD3 simpler baseline।
    • 
    


    

      Modern descendants: RED-Q, REDQ, MBPO — model-based-এর সাথে hybrid।
    


    

      DDPG raw এখন rarely use। TD3 ও SAC dominant। কিন্তু DDPG-এর core ideas (deterministic policy gradient, off-policy actor-critic) — সর্বত্র।
    


  


  

    
    
  


  

    
 ভাবনার প্রশ্ন


    

      প্র ০১Deterministic vs stochastic policy — fundamental advantage কোনটি?
      

        

          Deterministic ($\mu(s) = a$):
        

        
    
          
  • Pros:
            
      
              
    • Lower variance gradient (no sampling)।
      • 
              
    • Direct chain rule — backprop সরল।
      • 
              
    • Test deterministic — reproducible।
      • 
            
    
          
    • 
          
  • Cons:
            
      
              
    • No exploration — external noise add দরকার।
      • 
              
    • Multimodal optimal handle কঠিন।
      • 
              
    • Local optima trap risk।
      • 
            
    
          
    • 
        

        

          Stochastic ($\pi(a|s)$ — distribution):
        

        
    
          
  • Pros:
            
      
              
    • Built-in exploration।
      • 
              
    • Multimodal natural।
      • 
              
    • Risk-aware (entropy term)।
      • 
            
    
          
    • 
          
  • Cons:
            
      
              
    • Higher gradient variance।
      • 
              
    • Reparameterization trick দরকার।
      • 
            
    
          
    • 
        

        

          Empirical: MuJoCo continuous control-এ — both work। SAC slightly more robust due to entropy। TD3 simpler implementation।
        

        

          RLHF context: language generation-এ stochastic essential — multiple "valid" responses। deterministic বার বার একই text — useless।
        

      

    


    

      প্র ০২Why does target policy smoothing prevent overfitting?
      

        

          Target action smoothing: $\tilde{a}' = \mu_{\bar{\theta}}(s') + \epsilon$, $\epsilon \sim \mathcal{N}(0, \sigma)$ clipped।
        

        

          Insight:
        

        
    
          
  • Q-function-এর "narrow peaks" — overfit-প্রবণ।
    • 
          
  • actor সেই peak exploit করে — Q এত narrow যে নাড়লে immediately drop।
    • 
          
  • Smoothing — target এর "average over neighborhood" বানায়।
    • 
        

        

          Theoretical: "Quantile regression" perspective — soft target। SARSA expected Q-এর সাথে সংলগ্ন।
        

        

          Empirical: TD3 paper (Fujimoto 2018) ablation — smoothing remove → 30% performance drop।
        

        

          Connection to SAC: SAC's stochastic policy automatically smooth (sample noise)। TD3 explicitly add since deterministic।
        

      

    


    

      প্র ০৩Delayed actor update — কেন critic আগে stabilize?
      

        

          DDPG/TD3-এ — critic Q শিখছে, actor Q-এ optimize। ভুল Q → ভুল actor direction।
        

        

          Without delay:
        

        
    
          
  • Critic noisy estimate।
    • 
          
  • Actor সেই noisy Q চাসে।
    • 
          
  • Actor change → state distribution change → Q estimate further off।
    • 
          
  • Vicious cycle।
    • 
        

        

          With delay (every 2-d steps):
        

        
    
          
  • Critic d steps update — converge to current actor's true Q।
    • 
          
  • Then actor update — informed direction।
    • 
          
  • Stability significantly better।
    • 
        

        

          Trade-off: larger d → slower learning। typical d=2।
        

        

          Connection to TRPO: trust region idea — slow policy update, fast value update।
        

      

    


    

      প্র ০৪Real robot training — DDPG/TD3 vs SAC কোনটি practically suitable?
      

        

          Real robot — sample expensive, safety critical, sim-to-real challenging।
        

        

          SAC advantages real robot-এ:
        

        
    
          
  • Stochastic — natural exploration noise (no external)।
    • 
          
  • Entropy regularization — robust to model mismatch।
    • 
          
  • Auto-tune α — minimal tuning।
    • 
          
  • Higher sample efficiency typical।
    • 
        

        

          TD3 advantages:
        

        
    
          
  • Deterministic test time — predictable behavior।
    • 
          
  • Less compute — no entropy term।
    • 
          
  • Easier debug।
    • 
        

        

          Industry practice:
        

        
    
          
  • Boston Dynamics: control theory-নির্ভর, RL hybrid।
    • 
          
  • OpenAI dexterous hand: PPO + domain randomization।
    • 
          
  • Tesla Autopilot: imitation learning + RL fine-tune।
    • 
          
  • SAC academic papers ও research robots-এ popular।
    • 
        

        

          Recommendation:
        

        
    
          
  • Start with SAC default (Stable-Baselines3)।
    • 
          
  • Sim-to-real-এ domain randomization যোগ।
    • 
          
  • Real robot-এ start from imitation, RL fine-tune।
    • 
          
  • Always safety wrapper (action clip, fallback policy)।
    • 
        

      

    

  


  

    
 অনুশীলন

    
    
      
  1. 
        DPG gradient: $Q(s, a) = -|a - 2|^2$, $\mu_\theta(s) = \theta$, current $\theta = 5$। DPG actor gradient direction?
        
    
          
          
    
            
    $\nabla_a Q|_{a=5} = -2(5-2) = -6$। $\nabla_\theta \mu = 1$। Gradient = $-6 \cdot 1 = -6$।
    
            
    So $\theta$ decrease (toward optimal $a^*=2$)।
    
          
    
        
    
      
    2. 

      
  2. 
        TD3 target: $r=1, \gamma=0.99$, $Q_1(s', a') = 8, Q_2(s', a') = 6$, not done। target?
        
    
          
          
    
            
    $y = 1 + 0.99 \cdot \min(8, 6) = 1 + 5.94 = 6.94$।
    
            
    Single Q would have been $1 + 0.99 \cdot 8 = 8.92$ — overestimation।
    
          
    
        
    
      
    3. 

      
  3. 
        Code modify: উপরের TD3 update-এ — target action noise variance $\sigma$ fixed। linear decay over training-এর benefit?
        
    
          
          
    
            
    শুরুতে large noise — exploration। শেষে small — exploitation। Hyperparameter complexity বাড়ে কিন্তু performance improvement marginal। most paper-এ fixed σ default।
    
          
    
        
    
      
    4. 
    

  


  

    
 আরও পড়ুন

    
  







  
   এই পাঠে ভুল পেয়েছেন? জানান





  
    
 পূর্ববর্তী পাঠ

    
পাঠ ২২ · SAC