Policy Gradient theorem

১ · Value-based vs policy-based

Value-based (DQN): Q estimate, action = argmax Q।
Policy-based: $\pi_\theta(a|s)$ directly parameterize, gradient ascent on expected return।

২ · Objective function

Policy $\pi_\theta(a|s)$ — neural net। Expected return:

$$J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_t \gamma^t r_t \right] = \mathbb{E}_{\tau \sim \pi_\theta}[R(\tau)]$$

$\tau = (s_0, a_0, s_1, a_1, \ldots)$ — trajectory। সমস্যা: $J$-এর gradient compute করা সরাসরি কঠিন কারণ — distribution $p_\theta(\tau)$ also depends on $\theta$।

৩ · Log-derivative trick

Key identity:

$$\nabla_\theta p_\theta(\tau) = p_\theta(\tau) \cdot \nabla_\theta \log p_\theta(\tau)$$

Proof: $\nabla \log p = \nabla p / p \Rightarrow p \cdot \nabla \log p = \nabla p$।

কেন useful: $\nabla J = \int \nabla p_\theta(\tau) R(\tau) \, d\tau = \int p_\theta(\tau) \nabla \log p_\theta(\tau) R(\tau) \, d\tau = \mathbb{E}_{\tau \sim p_\theta}[\nabla \log p_\theta(\tau) R(\tau)]$।

Expectation form — Monte Carlo সম্ভব। trajectory sample, mean নিন।

৪ · Trajectory probability factorization

$$p_\theta(\tau) = p(s_0) \prod_t \pi_\theta(a_t | s_t) p(s_{t+1} | s_t, a_t)$$

$\log$ taking:

$$\log p_\theta(\tau) = \log p(s_0) + \sum_t \log \pi_\theta(a_t | s_t) + \sum_t \log p(s_{t+1} | s_t, a_t)$$

$\theta$-এ gradient — শুধু policy term:

$$\nabla_\theta \log p_\theta(\tau) = \sum_t \nabla_\theta \log \pi_\theta(a_t | s_t)$$

Environment dynamics (transition probability) — $\theta$-নির্ভর নয় — gradient-এ vanish। তাই model-free!

৫ · Policy gradient theorem

Combining:

$$\nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\left[ \left(\sum_t \nabla_\theta \log \pi_\theta(a_t | s_t)\right) R(\tau) \right]$$

Sutton et al. (1999) — fancier per-time-step form:

$$\nabla_\theta J(\theta) = \mathbb{E}_{s, a \sim \pi_\theta} \left[ \nabla_\theta \log \pi_\theta(a|s) \cdot Q^{\pi_\theta}(s, a) \right]$$

মানে — সব state-action pair-এ — log-prob-এর gradient × Q-value।

৬ · Variance reduction — baseline

Raw return $G$ — high variance। Subtract baseline $b(s)$:

$$\nabla J = \mathbb{E}\left[ \nabla \log \pi(a|s) \cdot (G - b(s)) \right]$$

Unbiased কেন: $\mathbb{E}_a[\nabla \log \pi(a|s) \cdot b(s)] = b(s) \cdot \nabla 1 = 0$।

Optimal baseline: $b(s) = V^\pi(s)$ — variance minimize। তখন $G - V = $ advantage $A$।

৭ · Score function estimator

$\nabla \log \pi(a|s)$ — "score function"। different policy parametrization:

• Softmax (discrete): $\pi(a|s) = \exp(\theta_a^T \phi(s)) / \sum_{a'} \exp(\theta_{a'}^T \phi(s))$।
• Gaussian (continuous): $\pi(a|s) = \mathcal{N}(\mu_\theta(s), \sigma^2)$।
• Beta: bounded continuous action।

৮ · PyTorch — log probability

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.distributions import Categorical, Normal

class CategoricalPolicy(nn.Module):
    """Discrete action policy (softmax)."""
    def __init__(self, state_dim, n_actions):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(state_dim, 128), nn.ReLU(),
            nn.Linear(128, n_actions),
        )
    def forward(self, s):
        logits = self.net(s)
        return Categorical(logits=logits)

class GaussianPolicy(nn.Module):
    """Continuous action policy (Gaussian)."""
    def __init__(self, state_dim, action_dim):
        super().__init__()
        self.shared = nn.Sequential(
            nn.Linear(state_dim, 128), nn.ReLU(),
        )
        self.mean = nn.Linear(128, action_dim)
        self.log_std = nn.Parameter(torch.zeros(action_dim))
    def forward(self, s):
        feat = self.shared(s)
        mu = self.mean(feat)
        std = self.log_std.exp().expand_as(mu)
        return Normal(mu, std)

# Sample action + log prob
state = torch.randn(1, 4)
policy = CategoricalPolicy(4, 2)
dist = policy(state)
action = dist.sample()
log_prob = dist.log_prob(action)
print(f"Action: {action.item()}, log π = {log_prob.item():.3f}")

৯ · Issues ও next steps

• High variance: Monte Carlo G — noisy। n-step বা TD estimate উন্নতি।
• On-policy: data current policy-র। old data reuse না — sample inefficient।
• Local optima: gradient ascent — local optimum-এ stuck হতে পারে।
• Step size sensitive: বড় step — collapse। ছোট — slow।

পরের পাঠ — REINFORCE (basic), Actor-Critic, A2C, PPO, SAC — এই issues address করার progressive uplift।