পাঠ ২১ · ৩২-এর মধ্যে · মডিউল ৩

Home / AI Courses / রিইনফোর্সমেন্ট লার্নিং / PPO

PPO — Proximal Policy Optimization

PPO — the workhorse of modern RL

৮ মিনিট পড়া উচ্চ · Advanced PyTorch

এই পাঠে যা শিখবেন

Clipped surrogate objective — derivation ও intuition
PPO-Clip vs PPO-Penalty (KL adaptive)
Full PyTorch implementation
RLHF-এ PPO-এর role

১ · TRPO থেকে PPO-এর পথ

TRPO theory ভাল কিন্তু implementation ৮০০+ lines। Schulman ভাবলেন — KL constraint-এর সরল alternative কী?

Idea: importance ratio-কে clip করো — extreme value reach না করুক। তাহলে — implicit "trust region"।

২ · Importance ratio

$$r_t(\theta) = \frac{\pi_\theta(a_t | s_t)}{\pi_{\theta_{old}}(a_t | s_t)}$$

$r_t = 1$ — policy unchanged। $r_t > 1$ — likely under new policy।

৩ · Clipped surrogate objective

$$L^{CLIP}(\theta) = \mathbb{E}_t \left[ \min\left( r_t(\theta) A_t, \, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) A_t \right) \right]$$

$\epsilon$ — clip parameter। typical $\epsilon = 0.2$।

৪ · Why this works — case analysis

Case A — $A_t > 0$ (good action):

$r_t < 1+\epsilon$ — full gradient flow।
$r_t > 1+\epsilon$ — clip stops gradient। policy বেশি probability assign করতে চাইলেও — না।

Case B — $A_t < 0$ (bad action):

$r_t > 1-\epsilon$ — full gradient flow (probability decrease)।
$r_t < 1-\epsilon$ — clip stops gradient। policy বেশি কমাতে চাইলেও — না।

Effect: ratio bounded in $[1-\epsilon, 1+\epsilon]$। policy old-এর কাছাকাছি — implicit trust region।

PPO clipping — extreme ratio gradient block। policy gradual update — collapse prevent।

৫ · PPO loss function — full

$$L_{total} = L^{CLIP} - c_1 \mathcal{L}^{VF} + c_2 \mathcal{H}[\pi]$$

Three terms:

$L^{CLIP}$ — clipped surrogate (maximize)।
$\mathcal{L}^{VF} = (V_\phi(s) - V_{target})^2$ — critic MSE (minimize)।
$\mathcal{H}[\pi]$ — entropy bonus (maximize)।

Default: $c_1 = 0.5, c_2 = 0.01$।

৬ · Multiple SGD epochs

TRPO single update। PPO — same data 4-10 epochs use। sample efficient।

Rollout — collect $T$ steps × $N$ envs।
Compute GAE advantages + value targets।
For $K$ epochs:
- Shuffle batch।
- Mini-batch SGD on $L^{CLIP} - c_1 \mathcal{L}^{VF} + c_2 \mathcal{H}$।
Discard data, repeat।

৭ · Full PyTorch PPO

Python · PPO

import gym
import torch
import torch.nn as nn
import torch.optim as optim
from torch.distributions import Categorical
import numpy as np

env = gym.make("CartPole-v1")
state_dim, n_actions = 4, 2
gamma, lam, eps_clip = 0.99, 0.95, 0.2
n_epochs, mb_size = 4, 64

class ActorCritic(nn.Module):
    def __init__(self):
        super().__init__()
        self.shared = nn.Sequential(nn.Linear(state_dim, 64), nn.Tanh(), nn.Linear(64, 64), nn.Tanh())
        self.actor = nn.Linear(64, n_actions)
        self.critic = nn.Linear(64, 1)
    def forward(self, s):
        f = self.shared(s)
        return Categorical(logits=self.actor(f)), self.critic(f).squeeze(-1)

model = ActorCritic()
opt = optim.Adam(model.parameters(), lr=3e-4)

def collect(T=2048):
    states, actions, log_probs, rewards, dones, values = [], [], [], [], [], []
    state, _ = env.reset()
    for _ in range(T):
        s_t = torch.FloatTensor(state)
        with torch.no_grad():
            dist, V = model(s_t)
            a = dist.sample()
            lp = dist.log_prob(a)
        next_state, r, term, trunc, _ = env.step(a.item())
        states.append(state); actions.append(a.item()); log_probs.append(lp.item())
        rewards.append(r); dones.append(float(term or trunc)); values.append(V.item())
        state = next_state
        if term or trunc: state, _ = env.reset()
    return states, actions, log_probs, rewards, dones, values

def compute_gae(rewards, values, dones, V_last):
    A, advs = 0, []
    V_next = V_last
    for t in reversed(range(len(rewards))):
        delta = rewards[t] + gamma * V_next * (1 - dones[t]) - values[t]
        A = delta + gamma * lam * (1 - dones[t]) * A
        advs.insert(0, A)
        V_next = values[t]
    return advs

for it in range(50):
    states, actions, old_lp, rewards, dones, values = collect(2048)
    with torch.no_grad():
        _, V_last = model(torch.FloatTensor(env.reset()[0]))
    advs = compute_gae(rewards, values, dones, V_last.item())
    returns = [a + v for a, v in zip(advs, values)]

    states_t = torch.FloatTensor(np.array(states))
    actions_t = torch.LongTensor(actions)
    old_lp_t = torch.FloatTensor(old_lp)
    advs_t = torch.FloatTensor(advs)
    advs_t = (advs_t - advs_t.mean()) / (advs_t.std() + 1e-8)
    returns_t = torch.FloatTensor(returns)

    # PPO multiple epochs
    for _ in range(n_epochs):
        idx = np.random.permutation(len(states))
        for i in range(0, len(idx), mb_size):
            mb = idx[i:i+mb_size]
            dist, V = model(states_t[mb])
            new_lp = dist.log_prob(actions_t[mb])
            ratio = torch.exp(new_lp - old_lp_t[mb])
            s1 = ratio * advs_t[mb]
            s2 = torch.clamp(ratio, 1-eps_clip, 1+eps_clip) * advs_t[mb]
            actor_loss = -torch.min(s1, s2).mean()
            critic_loss = (returns_t[mb] - V).pow(2).mean()
            entropy = dist.entropy().mean()
            loss = actor_loss + 0.5*critic_loss - 0.01*entropy
            opt.zero_grad(); loss.backward(); opt.step()

    if it % 5 == 0:
        print(f"Iter {it:3d} | rollout reward = {sum(rewards)/sum(dones):.1f}")

১০০ lines-এর কাছাকাছি — TRPO-এর ৮০০+ lines-এর তুলনায়। CartPole-এ ১০-২০ iteration-এ solve।

৮ · PPO-এর famous applications

OpenAI Five (Dota 2): ২০১৯-এ professional team-কে হারালো।
AlphaStar (StarCraft II): Grandmaster level।
Robot manipulation: OpenAI dexterous hand।
RLHF (ChatGPT): human preference থেকে LLM align।
Game AI: almost every recent RL game agent।

৯ · PPO best practices

$\epsilon = 0.2$ — most tasks।
4-10 epochs per batch।
2048-32768 timesteps per rollout।
Adam, learning rate 3e-4।
Advantage normalization।
Gradient clip (max norm 0.5)।
Linear learning rate decay।
Multiple seeds — RL high variance।

Cleanrl, Stable-Baselines3, RL Games — production implementations।

ভাবনার প্রশ্ন

প্র ০১PPO-এর "min" operator gradient-এর কী effect?

$L^{CLIP} = \min(r A, \text{clip}(r) A)$।

$A > 0$ case:

If $r < 1+\epsilon$: min = $rA$ (unclipped) — gradient flow।
If $r \ge 1+\epsilon$: min = $\text{clip}(r) A = (1+\epsilon)A$ (constant) — gradient zero।
Effect: probability বাড়াতে চাইলেও — clip-এ stop।

$A < 0$ case:

If $r > 1-\epsilon$: min = $rA$ — gradient flow (probability decrease)।
If $r \le 1-\epsilon$: min = $(1-\epsilon)A$ (constant) — gradient zero।
Effect: probability কমাতে চাইলেও — clip stop।

Asymmetry: clip এক direction-এ block, অন্য direction-এ allow। "trust region in spirit" — bad actions আরও বেশি bad করার চেষ্টা ও সুযোগ।

মূল উপলব্ধি: min ensures pessimistic clip — "policy collapse" risk eliminate। কিন্তু good direction-এ enough flexibility।

প্র ০২RLHF-এ PPO use — কেন আদর্শ choice?

ChatGPT-এর alignment pipeline-এ PPO crucial। কারণ:

(১) Per-token RL:

Generated text-এর প্রতি token = action।
Reward model থেকে scalar reward (sequence-level)।
PPO trajectory-style framework natural।

(২) KL constraint critical:

Base SFT model থেকে drift এড়ানো — nuisance text generation prevent।
PPO-এ extra KL term (RM reward + KL penalty)।

(৩) Multiple epochs:

Generation expensive (large LLM rollout)।
Same data বহুবার update — sample efficient।

(৪) Stability:

Clip prevents catastrophic update।
Large model-এ small step critical।

RLHF objective:

$L = L^{CLIP}(\theta) - \beta \cdot D_{KL}(\pi_\theta \| \pi_{SFT})$

Alternative — DPO (পাঠ ২৯): RL-পথ এড়িয়ে directly preference থেকে। সরল, কিন্তু কম flexible।

প্র ০৩PPO hyperparameters — কোনটি critical?

PPO অনেকগুলো knob-এর সাথে আসে। sensitivity ranking:

Most critical:

$\epsilon$ (clip): 0.1-0.3। 0.2 default ভাল most tasks।
Learning rate: 1e-4 to 3e-4। smaller for LLMs।
Rollout length: 2048 typical Atari, 256 small games।
Number of epochs: 4-10। বেশি — overfit, কম — underfit।

Moderately important:

$\gamma$: 0.99 default।
$\lambda$ (GAE): 0.95 default।
Mini-batch size: 64-256।

Less critical (default usually OK):

Entropy coef: 0.01।
Critic coef: 0.5।
Grad clip: 0.5।

Engelmann et al. (২০২২) "37 Implementation Details":

Subtle implementation tricks (advantage normalize, value clipping, ortho init) impact significant।
Code-level details often paper-এ undocumented।

Recommendation: production-grade PPO use Stable-Baselines3 বা cleanrl। reimplement করার চেয়ে।

প্র ০৪PPO-এর "implementation details" কেন original paper থেকে miss?

Engstrom et al. (2020) paper "Implementation Matters" famously showed — PPO performance largely from implementation tricks, not algorithm itself।

Key under-specified details:

Reward normalization/clipping।
Observation normalization (running mean/std)।
Value function clipping।
Orthogonal weight initialization।
Tanh activation (not ReLU)।
Adam epsilon (1e-5 not default 1e-8)।
Linear learning rate decay।
Advantage normalization scope।

Impact: ablation দেখা গেছে — এই tricks remove করলে PPO ভাঙে। algorithmic core (clipping) actually small contribution।

Why paper miss:

Researchers code optimize over time।
Submission focus on novel idea, not engineering।
Code release-এর সংস্কৃতি sparse ছিল 2017-এ।

আজকের best practice:

Code release mandatory many venues-এ।
Reproducibility checklist।
cleanrl-এর single-file implementation gold standard।

অনুশীলন

Clip output: $r=1.5$, $A=2$, $\epsilon=0.2$। PPO objective contribution?

$rA = 3$। $\text{clip}(1.5, 0.8, 1.2) \cdot 2 = 1.2 \cdot 2 = 2.4$। min = 2.4।

($A > 0$, $r > 1+\epsilon$ — clip activates)।
Negative case: $r=0.5$, $A=-1$, $\epsilon=0.2$। PPO?

$rA = -0.5$। $\text{clip}(0.5, 0.8, 1.2) \cdot (-1) = 0.8 \cdot (-1) = -0.8$। min = -0.8।

($A < 0$, $r < 1-\epsilon$ — clip activates, gradient block)।
Code modify: উপরের PPO-এ early stopping by KL divergence যোগ করুন (KL > 0.02 হলে stop)।
```
kl = (old_lp_t[mb] - new_lp).mean().item()
if kl > 0.02:
    break  # stop epoch — too far
```
OpenAI implementation এই trick — extra safety।