পাঠ ২১ · ৩২-এর মধ্যে · মডিউল ৩

Home / AI Courses / রিইনফোর্সমেন্ট লার্নিং / PPO

PPO — Proximal Policy Optimization

PPO — the workhorse of modern RL

৮ মিনিট পড়া উচ্চ · Advanced PyTorch

এই পাঠে যা শিখবেন

Clipped surrogate objective — derivation ও intuition
PPO-Clip vs PPO-Penalty (KL adaptive)
Full PyTorch implementation
RLHF-এ PPO-এর role

১ · TRPO থেকে PPO-এর পথ

TRPO theory ভাল কিন্তু implementation ৮০০+ lines. Schulman ভাবলেন — KL constraint-এর সরল alternative কী?

Idea: importance ratio-কে clip করো — extreme value reach না করুক। তাহলে — implicit "trust region".

২ · Importance ratio

$$r_t(\theta) = \frac{\pi_\theta(a_t | s_t)}{\pi_{\theta_{old}}(a_t | s_t)}$$

$r_t = 1$ — policy unchanged. $r_t > 1$ — likely under new policy.

৩ · Clipped surrogate objective

$$L^{CLIP}(\theta) = \mathbb{E}_t \left[ \min\left( r_t(\theta) A_t, \, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) A_t \right) \right]$$

$\epsilon$ — clip parameter. typical $\epsilon = 0.2$।

৪ · Why this works — case analysis

Case A — $A_t > 0$ (good action):

$r_t < 1+\epsilon$ — full gradient flow.
$r_t > 1+\epsilon$ — clip stops gradient. policy বেশি probability assign করতে চাইলেও — না।

Case B — $A_t < 0$ (bad action):

$r_t > 1-\epsilon$ — full gradient flow (probability decrease).
$r_t < 1-\epsilon$ — clip stops gradient. policy বেশি কমাতে চাইলেও — না।

Effect: ratio bounded in $[1-\epsilon, 1+\epsilon]$। policy old-এর কাছাকাছি — implicit trust region.

PPO clipping — extreme ratio gradient block. policy gradual update — collapse prevent.

৫ · PPO loss function — full

$$L_{total} = L^{CLIP} - c_1 \mathcal{L}^{VF} + c_2 \mathcal{H}[\pi]$$

Three terms:

$L^{CLIP}$ — clipped surrogate (maximize).
$\mathcal{L}^{VF} = (V_\phi(s) - V_{target})^2$ — critic MSE (minimize).
$\mathcal{H}[\pi]$ — entropy bonus (maximize).

Default: $c_1 = 0.5, c_2 = 0.01$।

৬ · Multiple SGD epochs

TRPO single update. PPO — same data 4-10 epochs use. sample efficient.

Rollout — collect $T$ steps × $N$ envs.
Compute GAE advantages + value targets.
For $K$ epochs:
- Shuffle batch.
- Mini-batch SGD on $L^{CLIP} - c_1 \mathcal{L}^{VF} + c_2 \mathcal{H}$।
Discard data, repeat.

৭ · Full PyTorch PPO

Python · PPO

import gym
import torch
import torch.nn as nn
import torch.optim as optim
from torch.distributions import Categorical
import numpy as np

env = gym.make("CartPole-v1")
state_dim, n_actions = 4, 2
gamma, lam, eps_clip = 0.99, 0.95, 0.2
n_epochs, mb_size = 4, 64

class ActorCritic(nn.Module):
    def __init__(self):
        super().__init__()
        self.shared = nn.Sequential(nn.Linear(state_dim, 64), nn.Tanh(), nn.Linear(64, 64), nn.Tanh())
        self.actor = nn.Linear(64, n_actions)
        self.critic = nn.Linear(64, 1)
    def forward(self, s):
        f = self.shared(s)
        return Categorical(logits=self.actor(f)), self.critic(f).squeeze(-1)

model = ActorCritic()
opt = optim.Adam(model.parameters(), lr=3e-4)

def collect(T=2048):
    states, actions, log_probs, rewards, dones, values = [], [], [], [], [], []
    state, _ = env.reset()
    for _ in range(T):
        s_t = torch.FloatTensor(state)
        with torch.no_grad():
            dist, V = model(s_t)
            a = dist.sample()
            lp = dist.log_prob(a)
        next_state, r, term, trunc, _ = env.step(a.item())
        states.append(state); actions.append(a.item()); log_probs.append(lp.item())
        rewards.append(r); dones.append(float(term or trunc)); values.append(V.item())
        state = next_state
        if term or trunc: state, _ = env.reset()
    return states, actions, log_probs, rewards, dones, values

def compute_gae(rewards, values, dones, V_last):
    A, advs = 0, []
    V_next = V_last
    for t in reversed(range(len(rewards))):
        delta = rewards[t] + gamma * V_next * (1 - dones[t]) - values[t]
        A = delta + gamma * lam * (1 - dones[t]) * A
        advs.insert(0, A)
        V_next = values[t]
    return advs

for it in range(50):
    states, actions, old_lp, rewards, dones, values = collect(2048)
    with torch.no_grad():
        _, V_last = model(torch.FloatTensor(env.reset()[0]))
    advs = compute_gae(rewards, values, dones, V_last.item())
    returns = [a + v for a, v in zip(advs, values)]

    states_t = torch.FloatTensor(np.array(states))
    actions_t = torch.LongTensor(actions)
    old_lp_t = torch.FloatTensor(old_lp)
    advs_t = torch.FloatTensor(advs)
    advs_t = (advs_t - advs_t.mean()) / (advs_t.std() + 1e-8)
    returns_t = torch.FloatTensor(returns)

    # PPO multiple epochs
    for _ in range(n_epochs):
        idx = np.random.permutation(len(states))
        for i in range(0, len(idx), mb_size):
            mb = idx[i:i+mb_size]
            dist, V = model(states_t[mb])
            new_lp = dist.log_prob(actions_t[mb])
            ratio = torch.exp(new_lp - old_lp_t[mb])
            s1 = ratio * advs_t[mb]
            s2 = torch.clamp(ratio, 1-eps_clip, 1+eps_clip) * advs_t[mb]
            actor_loss = -torch.min(s1, s2).mean()
            critic_loss = (returns_t[mb] - V).pow(2).mean()
            entropy = dist.entropy().mean()
            loss = actor_loss + 0.5*critic_loss - 0.01*entropy
            opt.zero_grad(); loss.backward(); opt.step()

    if it % 5 == 0:
        print(f"Iter {it:3d} | rollout reward = {sum(rewards)/sum(dones):.1f}")

১০০ lines-এর কাছাকাছি — TRPO-এর ৮০০+ lines-এর তুলনায়। CartPole-এ ১০-২০ iteration-এ solve.

৮ · PPO-এর famous applications

OpenAI Five (Dota 2): ২০১৯-এ professional team-কে হারালো।
AlphaStar (StarCraft II): Grandmaster level.
Robot manipulation: OpenAI dexterous hand.
RLHF (ChatGPT): human preference থেকে LLM align.
Game AI: almost every recent RL game agent.

৯ · PPO best practices

$\epsilon = 0.2$ — most tasks.
4-10 epochs per batch.
2048-32768 timesteps per rollout.
Adam, learning rate 3e-4.
Advantage normalization.
Gradient clip (max norm 0.5).
Linear learning rate decay.
Multiple seeds — RL high variance.

Cleanrl, Stable-Baselines3, RL Games — production implementations.

ভাবনার প্রশ্ন

প্র ০১PPO-এর "min" operator gradient-এর কী effect?

$L^{CLIP} = \min(r A, \text{clip}(r) A)$।

$A > 0$ case:

If $r < 1+\epsilon$: min = $rA$ (unclipped) — gradient flow.
If $r \ge 1+\epsilon$: min = $\text{clip}(r) A = (1+\epsilon)A$ (constant) — gradient zero.
Effect: probability বাড়াতে চাইলেও — clip-এ stop.

$A < 0$ case:

If $r > 1-\epsilon$: min = $rA$ — gradient flow (probability decrease).
If $r \le 1-\epsilon$: min = $(1-\epsilon)A$ (constant) — gradient zero.
Effect: probability কমাতে চাইলেও — clip stop.

Asymmetry: clip এক direction-এ block, অন্য direction-এ allow. "trust region in spirit" — bad actions আরও বেশি bad করার চেষ্টা ও সুযোগ।

মূল উপলব্ধি: min ensures pessimistic clip — "policy collapse" risk eliminate. কিন্তু good direction-এ enough flexibility.

প্র ০২RLHF-এ PPO use — কেন আদর্শ choice?

ChatGPT-এর alignment pipeline-এ PPO crucial. কারণ:

(১) Per-token RL:

Generated text-এর প্রতি token = action.
Reward model থেকে scalar reward (sequence-level).
PPO trajectory-style framework natural.

(২) KL constraint critical:

Base SFT model থেকে drift এড়ানো — nuisance text generation prevent.
PPO-এ extra KL term (RM reward + KL penalty).

(৩) Multiple epochs:

Generation expensive (large LLM rollout).
Same data বহুবার update — sample efficient.

(৪) Stability:

Clip prevents catastrophic update.
Large model-এ small step critical.

RLHF objective:

$L = L^{CLIP}(\theta) - \beta \cdot D_{KL}(\pi_\theta \| \pi_{SFT})$

Alternative — DPO (পাঠ ২৯): RL-পথ এড়িয়ে directly preference থেকে। সরল, কিন্তু কম flexible.

প্র ০৩PPO hyperparameters — কোনটি critical?

PPO অনেকগুলো knob-এর সাথে আসে। sensitivity ranking:

Most critical:

$\epsilon$ (clip): 0.1-0.3. 0.2 default ভাল most tasks.
Learning rate: 1e-4 to 3e-4. smaller for LLMs.
Rollout length: 2048 typical Atari, 256 small games.
Number of epochs: 4-10. বেশি — overfit, কম — underfit.

Moderately important:

$\gamma$: 0.99 default.
$\lambda$ (GAE): 0.95 default.
Mini-batch size: 64-256.

Less critical (default usually OK):

Entropy coef: 0.01.
Critic coef: 0.5.
Grad clip: 0.5.

Engelmann et al. (২০২২) "37 Implementation Details":

Subtle implementation tricks (advantage normalize, value clipping, ortho init) impact significant.
Code-level details often paper-এ undocumented.

Recommendation: production-grade PPO use Stable-Baselines3 বা cleanrl. reimplement করার চেয়ে।

প্র ০৪PPO-এর "implementation details" কেন original paper থেকে miss?

Engstrom et al. (2020) paper "Implementation Matters" famously showed — PPO performance largely from implementation tricks, not algorithm itself.

Key under-specified details:

Reward normalization/clipping.
Observation normalization (running mean/std).
Value function clipping.
Orthogonal weight initialization.
Tanh activation (not ReLU).
Adam epsilon (1e-5 not default 1e-8).
Linear learning rate decay.
Advantage normalization scope.

Impact: ablation দেখা গেছে — এই tricks remove করলে PPO ভাঙে। algorithmic core (clipping) actually small contribution.

Why paper miss:

Researchers code optimize over time.
Submission focus on novel idea, not engineering.
Code release-এর সংস্কৃতি sparse ছিল 2017-এ।

আজকের best practice:

Code release mandatory many venues-এ।
Reproducibility checklist.
cleanrl-এর single-file implementation gold standard.

অনুশীলন

Clip output: $r=1.5$, $A=2$, $\epsilon=0.2$। PPO objective contribution?

$rA = 3$। $\text{clip}(1.5, 0.8, 1.2) \cdot 2 = 1.2 \cdot 2 = 2.4$। min = 2.4.

($A > 0$, $r > 1+\epsilon$ — clip activates).
Negative case: $r=0.5$, $A=-1$, $\epsilon=0.2$। PPO?

$rA = -0.5$। $\text{clip}(0.5, 0.8, 1.2) \cdot (-1) = 0.8 \cdot (-1) = -0.8$। min = -0.8.

($A < 0$, $r < 1-\epsilon$ — clip activates, gradient block).
Code modify: উপরের PPO-এ early stopping by KL divergence যোগ করুন (KL > 0.02 হলে stop).
```
kl = (old_lp_t[mb] - new_lp).mean().item()
if kl > 0.02:
    break  # stop epoch — too far
```
OpenAI implementation এই trick — extra safety.